Construction site hidden danger identification method and system based on visual language large model

Abstract

The invention relates to a construction site hidden danger identification method and system based on a visual language large model. The method comprises the following steps: acquiring real-time acquired image data of high-place operation on a construction site; preprocessing the image data to obtain a to-be-detected image; inputting the to-be-detected image into the construction safety visual language model to obtain a hidden danger recognition result; the hidden danger recognition result comprises a high-altitude falling safety hidden danger type and standard natural language image subtitles. By adopting the method, intelligent technical support can be provided for safety management of the construction site, and the occurrence rate of high-altitude falling accidents is effectively reduced by identifying high-altitude falling hidden dangers of the construction site and generating standardized image subtitles.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence, and particularly relates to a construction site hidden danger identification method and system based on a visual language large model. BACKGROUND

[0002] With the development of artificial intelligence technology, image captioning technology has emerged, which realizes the generation of readable text description for visual input by fusing the semantics of images and natural language, and assists in judging the image content features.

[0003] The existing image captioning technology has obvious limitations in high-fall hidden danger identification in the construction site. High-fall hidden dangers usually involve multi-dimensional features of personnel behavior and object state. The visual language model (VLM) implementing the image captioning technology can usually only identify a single type of hidden danger, lacks deep understanding and semantic expression of the hidden danger situation in the image, and is difficult to comprehensively capture the combination of high-fall hidden dangers in the construction site. In addition, the existing image captioning VLM model often lacks unified standards for describing the same safety hidden danger, and the expression is diverse, which not only increases the complexity of construction safety management, but also may cause the model to fail to accurately identify due to the difference in the expression of key hidden danger features, thereby causing the missed detection of safety hidden dangers and reducing the efficiency of construction safety management. SUMMARY

[0004] Therefore, it is necessary to provide a construction site hidden danger identification method and system based on a visual language large model, which can automatically generate standardized description text from the high-work scene information in the construction site image and give hidden danger labels.

[0005] In a first aspect, the application provides a construction site hidden danger identification method based on a visual language large model, comprising:

[0006] obtaining image data of real-time collected high-work in the construction site;

[0007] preprocessing the image data to obtain a to-be-detected image;

[0008] inputting the to-be-detected image into a construction safety visual language model to obtain a hidden danger identification result; the hidden danger identification result includes a high-fall safety hidden danger type and a standardized natural language image caption.

[0009] In one embodiment, the construction safety visual language model obtains the hidden danger identification result by the following method, comprising:

[0010] performing multi-scale language feature extraction on the to-be-detected image to obtain a multi-scale feature vector; the multi-scale feature vector includes a human pose, a scene structure, and an object state;

[0011] The high-fall safety hazard type is predicted according to the multi-scale feature vector;

[0012] The multi-scale feature vector is semantically aligned and mapped according to the high-altitude operation construction safety standard term database and the term constraint prompt, and a standard natural language image caption is obtained.

[0013] The hazard identification result is constructed according to the high-fall safety hazard type and the standard natural language image caption.

[0014] In one of the embodiments, the high-altitude operation construction safety standard term database is constructed by the following method:

[0015] The target construction safety standard is obtained, and the high-fall hazard related terms are extracted, to obtain a plurality of standard terms and their corresponding synonyms and near synonyms;

[0016] According to the standard terms and their corresponding synonyms and near synonyms, combined with the high-altitude operation normal state standard terms and their natural language descriptions corresponding to the safety standard terms, a high-altitude operation construction safety standard term database is constructed using a bidirectional comparison structure.

[0017] In one of the embodiments, the construction safety visual language model is obtained by the following method:

[0018] Image data of high-altitude operation in the construction site is obtained, preprocessed and text labeled to obtain an image-text pair dataset; the image-text pair dataset includes images and their corresponding safety hazard types and a plurality of text descriptions;

[0019] The image-text pair dataset is converted into a standardized training dataset by a natural language description standard paradigm;

[0020] The pre-trained visual language model is structurally adjusted using a lightweight adapter of the MS-Swift framework and a low-rank matrix, and task-oriented multi-task joint training is performed according to the standardized training dataset to obtain a construction safety visual language model.

[0021] In one of the embodiments, image data of high-altitude operation in the construction site is obtained, preprocessed and text labeled to obtain an image-text pair dataset, including:

[0022] The image similarity of the image data is calculated using a perceptual hashing algorithm, and the images with image similarity exceeding a preset threshold are saved by selecting one, to obtain a deduplicated image set;

[0023] The deduplicated image set is screened by resolution, and low-quality images are removed to obtain a target image set;

[0024] According to the experience of experts, the target image set is annotated for safety hazards to obtain the safety hazard types and text descriptions corresponding to each image; the safety hazard types include normal conditions, unsafe behaviors of people and unsafe states of objects;

[0025] According to the images and the safety hazard types and text descriptions corresponding thereto, an image-text pair data set is obtained.

[0026] In one of the embodiments, the image-text pair data set is converted into a natural language description standard paradigm to obtain a standardized training data set, including:

[0027] Key information of the text description of the image-text pair data set is identified based on a BERT pre-trained language model;

[0028] The key information is mapped into a high-altitude work construction safety specification standard term database, and a design rule engine is used to process complex text that cannot be directly mapped to obtain corresponding standard terms;

[0029] The standard terms are used to replace the corresponding text descriptions in the image-text pair data set to obtain the standardized training data set.

[0030] In one of the embodiments, a lightweight adapter and a low-rank matrix of an MS-Swift framework are used to adjust the structure of a pre-trained visual language model, and task-oriented multi-task joint training is performed according to the standardized training data set to obtain a construction safety visual language model, including:

[0031] Based on a minimization log-likelihood loss fine-tuning target, the main weight of the pre-trained visual language model is frozen using a lightweight adapter and a low-rank matrix of an MS-Swift framework, and the parameters of an image encoder and a text encoder are optimized to obtain an updated weight matrix;

[0032] Simulated text noise is added to the standardized training data set to obtain enhanced training data;

[0033] Based on a minimization joint loss training target, text prompts and dynamic term constraints, the enhanced training data is used to perform multi-task joint training on the visual language model corresponding to the updated weight matrix to obtain model parameters; the multi-task corresponds to safety hazard type classification and specification natural language image caption generation;

[0034] The construction safety visual language model is constructed according to the model parameters;

[0035] The log-likelihood loss is obtained by the following formula:

[0036]

[0037] Wherein, L is the log-likelihood loss; P(C i|I i ; θ) is the probability of generating a canonical natural language image caption C i given the training data set I i and the model parameters θ, which contains the updated weight matrix W'; K is the total number of data samples of the training data set.

[0038] In one embodiment, the parameters of the image encoder and the text encoder are optimized to obtain the updated weight matrix based on the fine-tuning target of minimizing the log-likelihood loss, using the lightweight adapter of the MS-Swift framework and the low-rank matrix to freeze the main body weights of the pre-trained visual language model, including:

[0039] The updated weight matrix is obtained by the following formula:

[0040]

[0041] where W ' is the updated weight matrix; W is the main body weight matrix, and B is a low-rank matrix, and d and k are the matrix dimensions of W; r is the rank of the lightweight adapter of the MS-Swift framework, and r << min(d, k); α is a scaling parameter for adjusting the influence of the updated weight matrix on the main body weight matrix.

[0042] In one embodiment, the method further comprises:

[0043] The evaluation result is obtained by comparing the canonical natural language image caption generated by the construction safety visual language model with the corresponding text description in the image-text pair data set through multiple evaluation indicators, including word accuracy, semantic alignment degree and syntactic structure;

[0044] The model parameters of the construction safety visual language model are adjusted according to the evaluation result.

[0045] In a second aspect, the application also provides a construction site hidden danger identification system based on a visual language large model, comprising:

[0046] A data acquisition module is configured to acquire image data of high-altitude work in a construction site collected in real time;

[0047] A data processing module is configured to pre-process the image data to obtain a to-be-detected image;

[0048] A model algorithm module is configured to input the to-be-detected image into a construction safety visual language model to obtain a hidden danger identification result, wherein the hidden danger identification result includes a high-altitude falling safety hidden danger type and a canonical natural language image caption.

[0049] The construction site hidden danger identification method and system based on the visual language large model can realize automatic processing from image input to hidden danger type and standard text output, significantly improve the understandability and data usability of the identification result, and are especially suitable for construction supervision scenarios that require text description, realize early identification and early intervention of high-altitude operation hidden dangers, effectively reduce the probability of high-altitude falling accidents, and provide intelligent and forward-looking technical support for building safety management. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiment or related art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0051] Figure 1 The flowchart of the construction site hidden danger identification method based on the visual language large model of the present application;

[0052] Figure 2 The construction safety visual language model construction flowchart of the present application;

[0053] Figure 3 The sub-step flowchart of step S202;

[0054] Figure 4 The composition structure diagram of the construction site hidden danger identification system based on the visual language large model of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0056] In one embodiment, as shown in Figure 1 , a construction site hidden danger identification method based on a visual language large model is provided. In this embodiment, the method is applied to a terminal for illustration. It should be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and can be realized through the interaction of the terminal and the server. In this embodiment, the method includes the following steps:

[0057] S101, acquire image data of high-altitude operation in a construction site in real time.

[0058] Illustratively, the image data can be collected by fixed high-definition cameras or mobile wearable devices deployed in key high-altitude operation areas of the construction site. Considering the characteristics of the construction site, such as varying lighting conditions, various operation levels, and random shooting angles, the acquisition system needs to have basic functions such as automatic exposure adjustment, night enhancement, and image stabilization to ensure that the image quality meets the analysis requirements. Optionally, to meet the requirements of privacy protection and data compliance, a face blurring processing mechanism is introduced during the acquisition process. For the face information that may be contained in the image, real-time or batch image desensitization algorithms are used to automatically blur the image, ensuring that the original image data does not leak personal identity features, thereby meeting the requirements of data security regulations and ethics.

[0059] S102, pre-process the image data to obtain a to-be-detected image.

[0060] Illustratively, the pre-processing mainly includes image size normalization, noise suppression, and format conversion. Among them, size normalization is used to uniformly adjust the image to the standard resolution accepted by the model to avoid feature extraction deviation caused by input size differences; noise suppression eliminates image noise introduced by factors such as on-site lighting, smoke, etc. through image enhancement methods such as Gaussian filtering or bilateral filtering; format conversion ensures that the image encoding method is consistent with the input interface of the visual language model.

[0061] S103, input the to-be-detected image into a construction safety visual language model to obtain a hazard identification result; the hazard identification result includes a high-altitude falling safety hazard type and a standard natural language image caption.

[0062] Illustratively, the processed image is input into the construction safety visual language model (CS-VLM) to automatically complete the hazard identification and caption generation tasks. Illustratively, CS-VLM is based on Qwen2.5-7B-Instruct and completes exclusive adaptation to the construction safety field through low-rank matrix adaptation technology (LoRA). Specifically, CS-VLM uses its internal visual encoder to extract semantic features of the input image, identifying multi-dimensional information such as structural components, human poses, operation behaviors, and environmental states in the image. Further, the visual features are input into the language decoding module to generate a natural language caption with domain terminology constraints. This caption not only accurately describes the operation state in the image, but also uses the standard expressions in the construction safety standard terminology system to ensure the professionalism and readability of the output.

[0063] Meanwhile, the CS-VLM can also complete the class recognition task of the high-fall hidden danger in parallel through its multi-task learning structure on the basis of natural language description generation, and output corresponding structured hidden danger type labels, such as, for example, not wearing a safety belt, missing edge protection, non-standard scaffold erection, etc. Optionally, the hidden danger recognition result can be transmitted to a construction safety information platform or an edge early warning terminal simultaneously, so as to realize real-time safety state perception and early warning notification of the target operation area.

[0064] In the above construction site hidden danger recognition method based on the visual language large model, the real-time collected construction site high-altitude operation image data can be acquired to realize real-time monitoring of the construction site high-altitude operation state and improve the timeliness of hidden danger discovery. The image data is subjected to normalization, filtering, format conversion and other preprocessing operations to generate image samples suitable for model input, improve recognition accuracy and stability, avoid interference of low-quality images on model performance, enhance robustness to differences in shooting conditions under actual construction site environment, unify image format for efficient model loading and streaming processing, and improve hidden danger recognition operation efficiency and compatibility. The pre-trained CS-VLM model is used to complete image semantic understanding and output hidden danger recognition results. The output natural language subtitles are the results of term standardization, which are convenient for supervision, alarm recording and standard archiving, and realize a complete conversion path from image perception to semantic understanding, improving the explainability and human-computer interaction friendliness of the model output.

[0065] In one of the embodiments, the construction safety visual language model obtains the hidden danger recognition result by the following method, which includes:

[0066] S11. Multi-scale language feature extraction is performed on the to-be-detected image to obtain a multi-scale feature vector; the multi-scale feature vector includes a human pose, a scene structure and an object state.

[0067] Illustratively, the CS-VLM model performs multi-scale semantic feature extraction on the input to-be-detected image. Specifically, the CS-VLM model uses a Transformer (attention mechanism-based deep learning model) architecture to uniformly model the human pose, scene structure and object state in the image, and obtain a multi-scale feature vector with spatial hierarchy and semantic association. The multi-scale processing method allows the model to focus on both local details and global context, enabling it to identify high-fall hidden dangers such as a safety belt hanging loosely but not tied, which requires the integration of background and action state for judgment.

[0068] S12. The high-fall safety hidden danger type is predicted according to the multi-scale feature vector.

[0069] Illustratively, the CS-VLM model completes the prediction of the high-fall safety hazard type based on a multi-scale feature vector. Specifically, to achieve multi-task parallel optimization, the CS-VLM model adds a classification and discrimination branch at the decoding end. Through joint optimization of the text generation and hazard classification tasks in the training stage, the model can output a structured safety hazard category label at one time in the inference stage. Optionally, the safety hazard category label can correspond to a specific risk situation, such as the absence of edge protection, abnormality of a hanging basket safety device, etc., or it can be a coarse classification label for normal situations, unsafe human behavior, and unsafe object state.

[0070] S13, based on the high-fall construction safety standard term database and the term constraint prompt, performing semantic alignment and term mapping on the multi-scale feature vector to obtain a standard natural language image caption.

[0071] Illustratively, based on the high-fall construction safety standard term database and the term prompt mechanism constructed therefrom, the multi-scale feature vector is subjected to semantic feature generation. In the fine-tuning training stage, the CS-VLM model has learned to automatically call the term table when generating descriptions through a large number of standard term annotation samples, ensuring that the description content is consistent with the construction safety standard expression. Specifically, the CS-VLM model matches the corresponding standard term structure in the database according to the generated image semantic vector, and generates a unique, standard, and professional natural language caption through the decoder.

[0072] S14, constructing a hazard identification result according to the high-fall safety hazard type and the standard natural language image caption.

[0073] Illustratively, the high-fall safety hazard type and the generated standard natural language image caption are bound to construct a complete hazard identification result.

[0074] In one embodiment, the high-fall construction safety standard term database is constructed by the following method:

[0075] S21, obtaining a target construction safety standard and extracting high-fall hazard-related terms to obtain a plurality of standard terms and their corresponding synonyms and near-synonyms.

[0076] Illustratively, high-fall hazard-related terms are extracted from the current national and local construction safety standard documents such as the “Technical Code for Safety of High-Fall Construction” JGJ80-2016 and the “General Specification for Construction Scaffolding” GB 55023-2022, and synonyms, near-synonyms, and commonly used variant expressions of each term are obtained to ensure coverage of common description methods for high-fall construction work.

[0077] S22, according to each standard term and its corresponding synonym and near synonym, combined with the corresponding high-altitude operation normal state standard term of safety specification term and its natural language description, a high-altitude operation construction safety specification standard term database is constructed using a bidirectional comparison structure.

[0078] The standard term is combined with the synonym and near synonym that is easy to appear in the normal operation of the construction site to construct a term database with a bidirectional comparison structure. As shown in Table 1, each hidden danger standard term corresponds to one or more expression modes.

[0079] The high-altitude operation construction safety specification standard term database is not only used as a subtitle standardization target in the CS-VLM model training and application stage, but also provides prompt (prompt word) prompt and vocabulary constraint mechanism in model fine-tuning, ensuring that the CS-VLM model can accurately call standard terms when generating natural language descriptions, thereby improving the professionalism, consistency and semantic accuracy of the output results.

[0080] Table 1 Example of high-altitude operation construction safety specification standard term database

[0081]

[0082] In one embodiment, as shown in Figure 2 , the construction safety visual language model is obtained by the following method:

[0083] S201, obtaining image data of high-altitude operation at the construction site, and pre-processing and text labeling to obtain image-text pair data set; the image-text pair data set includes images and their corresponding safety hazard types and multiple text descriptions.

[0084] Illustratively, high-altitude operation image data from different construction sites is widely collected. Data collection covers various construction situations, including scaffold erection, stair passage erection, basket operation, and high-risk scenarios such as exterior wall decoration, to ensure the semantic diversity and situational representativeness of the training data. In the image acquisition and processing stage, an image desensitization mechanism is introduced uniformly, and through face detection and blurring algorithm, the face information of workers in the image is blurred to ensure that the collected images do not contain identifiable personal identity features, thereby constructing an image sample library that meets the data compliance standards.

[0085] After obtaining the image data, a team of safety experts with high-altitude work experience is organized to conduct detailed labeling on each image, providing multiple natural language descriptions, and manually confirming the types of high-altitude fall safety hazards involved. For example, each image corresponds to 5 independent descriptions, covering unsafe behaviors, unsafe states of objects, and environmental unsafe factors, ensuring that the model training phase can fully learn the semantic features of the hazard situation and form an original image-text pair dataset, which includes images, corresponding multiple natural language subtitles, and manually annotated safety hazard category labels.

[0086] S202, converting the image-text pair dataset to a natural language description standard paradigm to obtain a standardized training dataset.

[0087] Illustratively, the natural language description text in the image-text pair dataset is standardized to eliminate semantic differences caused by diverse expressions. Specifically, based on the BERT (Bidirectional Encoder Representation from Transformers) pre-trained language model, the semantic analysis of each description is performed to identify key information entities such as "safety belt", "not hanging", "missing edge guardrail", etc. and combined with a pre-built high-altitude work construction safety specification terminology database, the rule engine is used to map the key word groups in the description text to standard terminology structures, handling complex language situations such as polysemy expressions, structural ambiguity or omitted descriptions, ensuring that each image only retains a unique and industry-standard natural language description, and keeping it consistent with the original hazard category annotation.

[0088] S203, using a lightweight adapter of the MS-Swift framework and a low-rank matrix to adjust the structure of the pre-trained visual language model, and performing task-oriented multi-task joint training according to the standardized training dataset to obtain a construction safety visual language model.

[0089] Illustratively, the lightweight adaptation technology under the MS-Swift framework is used in combination with the LoRA structure to adjust the structure and task-specific fine-tuning of the pre-trained visual language model Qwen2.5-7B-Instruct. Specifically, to avoid the memory resource pressure and training time cost caused by full model parameter update, in the multi-head attention module of the Qwen2.5-7B-Instruct Transformer architecture, the three weight matrices W q , W k , W vThe update with low intrinsic rank is performed. Illustratively, the training and verification of the complete model are completed on a GPU (graphics processor) with a 24 GB memory, and the training time is further shortened and the engineering usability is improved when multiple GPUs are parallel.

[0090] Further, task-oriented data augmentation and multi-task joint training mechanism are adopted to optimize the model's caption generation ability and hazard type discrimination ability. In terms of data augmentation, a description disturbance mechanism is introduced during the training stage, including synonym replacement, sentence restructuring, irrelevant information insertion, etc., to simulate language noise in actual input and improve the model's robustness to non-standard text. In terms of task objectives, the model optimizes two output branches simultaneously, i.e., natural language caption generation and safety hazard classification. The loss function is the weighted sum of the CIDEr loss of the caption generation task and the cross-entropy loss of the classification task, ensuring that the model can output structured category labels and natural language standard descriptions simultaneously, achieving the unity of semantic consistency and pragmatic adaptability.

[0091] In one of the embodiments, image data of high-altitude work at the construction site is acquired, preprocessed and text-labeled to obtain an image-text pair dataset, including:

[0092] S31, the perceptual hashing algorithm is used to calculate the image similarity of the image data, and the images with image similarity exceeding a preset threshold are saved alternatively to obtain a deduplicated image set.

[0093] Illustratively, the perceptual hashing algorithm (Perceptual Hashing, pHash) is used to calculate the similarity of the visual content of the image data. Specifically, the perceptual hashing algorithm generates a hash fingerprint by extracting the structural features of the image, and compares the Hamming distance between images to achieve fast and robust similarity judgment. Illustratively, when the similarity between images exceeds the preset threshold of 95%, only one of the images is retained to avoid sample redundancy and bias caused by duplicate images during model training.

[0094] S32, the resolution of the deduplicated image set is screened, and low-quality images are removed to obtain a target image set.

[0095] Illustratively, according to the required detail visibility of high-altitude fall hazard identification, the image resolution lower limit is set to 1024x768, and images below this threshold are removed. Further, images that are blurred, overexposed or severely obstructed are removed, and the screened images constitute the target image set that meets the quality standards.

[0096] S33, the target image set is labeled for safety hazards according to expert experience to obtain the safety hazard types and text descriptions corresponding to each image. The safety hazard types include normal conditions, unsafe behaviors of people and unsafe states of objects.

[0097] Illustratively, invite construction safety experts with high work experience to conduct semantic annotation on the target image set, each image is assigned a type of high fall safety hazard type, and several corresponding natural language descriptions are provided. The safety hazard type covers normal situation, unsafe behavior of people and unsafe state of things. Illustratively, the normal situation can be that the safety belt is hung in standard, and the outer frame is erected in accordance with the standard; the unsafe behavior of people can be that the worker does not wear a safety belt, and the worker leans against the edge protection; the unsafe state of things can be that the railing is broken, and the safety net is damaged. The experts refer to the field operation habit, accident warning elements and standard terms to conduct multi-dimensional semantic expression on the image content, and construct a preliminary image-text pair data.

[0098] S34, according to each image and its corresponding safety hazard type and text description, an image-text pair data set is obtained.

[0099] The image and its corresponding safety hazard type and text description are paired to form an original image-text pair data set.

[0100] In one embodiment, as shown in Figure 3 The image-text pair data set is converted into a natural language description standard paradigm to obtain a standardized training data set, including:

[0101] S301, based on the BERT pre-training language model, the key information of the text description of the image-text pair data set is identified.

[0102] Illustratively, based on the BERT pre-training language model, the semantic analysis of each natural language description is performed, and the key information including objects, behaviors and state judgment words is automatically identified. The BERT model relies on its context understanding ability, and can extract the semantic core from the diversified sentence expressions, providing semantic anchor points for subsequent mapping and reconstruction.

[0103] S302, the key information is mapped to the high work construction safety standard term database, and the complex text that cannot be directly mapped is processed using a design rule engine to obtain the corresponding standard term.

[0104] The key information is mapped to the high work construction safety standard term database to obtain the corresponding standard term, and in the case that cannot be directly mapped, a pre-defined rule engine is called to analyze and complete the complex structure, as shown in Table 2, the rule engine supports processing of language ambiguity such as omitted description, parallel relationship and omitted subject, to ensure that the generated text is semantically complete and expressed in accordance with the standard term specification.

[0105] Table 2: Example of complex rule engine for high work image natural language description standard paradigm conversion

[0106]

[0107] S303, replace the corresponding text description in the image-text pair dataset with standard terms to obtain a standardized training dataset.

[0108] Illustratively, according to the mapping result and the rule engine output, the original description is replaced with standard terms to form a natural language subtitle with consistent structure and standardized semantics. Each image finally corresponds to a unique, standardized text description, which is input into the CS-VLM model as a training target, thereby significantly improving the consistency, professionalism and interpretability of the model output description.

[0109] The above method develops a natural language description standard paradigm conversion technology, converts diverse subtitle descriptions into a unified and standardized expression, improves the accuracy and consistency of the description, and obtains a standardized dataset that not only optimizes the language generation learning process of the model, but also avoids the influence of description ambiguity and semantic drift on the performance of the model, providing a solid data foundation for the stability and generalization ability of the model training.

[0110] In one embodiment, a lightweight adapter of the MS-Swift framework and a low-rank matrix are used to adjust the structure of the pre-trained visual language model, and task-oriented multi-task joint training is performed according to the standardized training dataset to obtain a construction safety visual language model, including:

[0111] S41, based on the minimization of the log-likelihood loss, the main weight of the pre-trained visual language model is frozen using the lightweight adapter of the MS-Swift framework and the low-rank matrix, and the parameters of the image encoder and the text encoder are optimized to obtain an updated weight matrix.

[0112] The log-likelihood loss is obtained by the following formula:

[0113]

[0114] Where L is the log-likelihood loss; P(C i |I i ; θ) is the probability of generating a standardized natural language image caption C i given the training dataset I i and the model parameters θ, which contains the updated weight matrix W'; K is the total number of data samples in the training dataset.

[0115] Illustratively, in order to reduce the computing resources required for large model fine-tuning and avoid the risk of overfitting, the main parameters of the pre-trained model are frozen, and only the parameters of the key modules are optimized. Specifically, a low-rank matrix structure is inserted in the interaction layer between the visual encoder and the text decoder of Qwen2.5-7B-Instruct, and a lightweight adapter under the MS-Swift framework is used to control the parameter update path. The inserted structure adopts the form of LoRA, and the update weight matrix is constructed by inserting a small additive weight into the original linear weight matrix of the Qwen2.5-7B-Instruct model. The additive weight maintains the dimension of the original linear weight matrix. Under the premise of keeping the original linear weight matrix frozen, only the inserted part is optimized, so that the Qwen2.5-7B-Instruct model can efficiently capture the semantic features related to the safety hazards of high-fall. The goal of fine-tuning is to minimize the log-likelihood loss function, that is, to maximize the probability of the model generating a standard natural language caption under the given input image and known semantic conditions.

[0116] S42, add simulated text noise to the standardized training data set to obtain enhanced training data.

[0117] In order to improve the language robustness of the model in complex actual environment, a task-oriented data enhancement mechanism is introduced. Illustratively, on the basis of the standardized training data set, the noise in the text is simulated by randomly replacing words, the diversity of the text is simulated by sentence reorganization, and the interference content in the text is simulated by adding irrelevant information, thereby enhancing the robustness of the CS-VLM model to complex text scenarios. The enhanced samples can guide the model to learn to generate stable output of standard terminology even in the presence of noise, thereby improving the anti-interference ability.

[0118] S43, based on the minimum joint loss training target, the text prompt and the dynamic term constraint, the enhanced training data is used to perform multi-task joint training on the visual language model corresponding to the updated weight matrix to obtain model parameters; the multi-task corresponds to safety hazard type classification and standard natural language image caption generation.

[0119] In combination with the text prompt and the dynamic constraint mechanism of the term, multi-task joint training is performed under the enhancement of training data driving, that is, the classification of high-fall safety hazards and the generation of text are simultaneously optimized. Illustratively, a classification head is added to the output layer of the CS-VLM model to predict the category of high-fall safety hazards, and a joint loss function is used to balance the performance of the text generation and high-fall safety hazard classification tasks. Illustratively, the language generation loss is represented by the CIDEr loss, and the classification loss is represented by the cross-entropy loss. The two are weighted and summed by the joint loss function to form a multi-task optimization objective. During the training process, the CS-VLM model outputs standardized natural language descriptions and safety hazard classification results by sharing the image semantic encoder, thereby achieving the unification of semantic understanding and structural output.

[0120] S44, constructing a construction safety visual language model according to the model parameters.

[0121] Illustratively, the optimal parameters obtained by training are loaded back into the model structure to formally construct the construction safety visual language model CS-VLM.

[0122] The above method uses the lightweight adapter of the MS-Swift framework and the low-rank matrix technology to efficiently fine-tune the pre-trained visual language model, optimize the model performance, and significantly improve the recognition ability of high-fall hazards in complex construction scenes.

[0123] In one embodiment, based on the minimization of the log-likelihood loss fine-tuning target, the lightweight adapter of the MS-Swift framework and the low-rank matrix are used to freeze the main weight of the pre-trained visual language model, optimize the parameters of the image encoder and the text encoder, and obtain the updated weight matrix, including:

[0124] The updated weight matrix is obtained by the following formula:

[0125]

[0126] where W' is the updated weight matrix; W is the main weight matrix, and B is a low-rank matrix, and d and k are the matrix dimensions of W; r is the rank of the lightweight adapter of the MS-Swift framework, and r << min(d, k); a is a scaling parameter used to adjust the influence of the updated weight matrix on the main weight matrix.

[0127] Illustratively, during the domain adaptation fine-tuning process, the update of the weight matrix has a low intrinsic rank. That is, for the main weight matrix A trainable low-rank matrix B is inserted into the partially linear layer of the model structure and r<<min(d, k), forming a structure of W' = W + BA. d and k are the dimensions of W, and r is the rank of the adapter. In the fine-tuning process, the original main body weight matrix W is kept frozen, and only the inserted part is optimized. In the weight updating process of the low-rank matrices A and B, B is initialized to zero, and A is initialized with small random values from a Gaussian distribution. The updated matrix is a structure, where a is a scaling parameter for adjusting the influence of the new weight on the original main body weight. Further, the above fine-tuning method is applied to the three weight matrices W q , W k , W v .

[0128] In one of the embodiments, the method further comprises:

[0129] S51, compare the specification natural language image caption generated by the construction safety visual language model with the corresponding text description in the image-text pair data set through multiple evaluation indexes, and obtain an evaluation result; the evaluation indexes include word accuracy, semantic alignment degree and syntax structure.

[0130] Illustratively, the evaluation stage compares the natural language image captions generated by the CS-VLM on the test set with the texts in the image-text pair dataset, and evaluates the language generation quality using mainstream indicators. Among them, BLEU (Bilingual Evaluation Understudy) evaluates the word accuracy of the generated captions by calculating the n-gram matching accuracy between the generated captions and the reference captions. ROUGE_L (Recall-Oriented Understudy for Gisting Evaluation) calculates the similarity between the generated captions and the reference captions based on the longest common subsequence (LCS), emphasizing the matching of the order and structure. METEOR (Metric for Evaluation of Translation with Explicit Ordering) considers precision, recall, synonym matching and stem reduction to evaluate the semantic consistency of the generated captions and the reference captions. SPICE (Semantic Propositional Image Caption Evaluation) analyzes the semantic scene graph of the generated captions and the reference captions to evaluate the expression ability of objects, attributes and relationships. CIDEr (Consensus-based Image Description Evaluation) is based on the n-gram matching of the term frequency-inverse document frequency (TF-IDF) weighting to evaluate the consistency of the generated captions and the reference captions in terms of semantic consensus.

[0131] S52, adjusting the model parameters of the construction safety visual language model according to the evaluation result.

[0132] According to the evaluation result, the model parameters are further fine-tuned. Illustratively, if the recognition accuracy of a certain category or the generation accuracy of a certain term does not meet the standard, the corresponding sample can be traced back, the term prompt template can be adjusted or the loss weight can be adjusted, forming a closed-loop optimization mechanism to ensure that the final performance of the model meets the actual deployment requirements.

[0133] It should be understood that although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the embodiments described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0134] Based on the same inventive concept, the embodiments of the present application also provide a visual language-based large model construction site hazard identification system for implementing the visual language-based large model construction site hazard identification method described above. The implementation scheme for solving problems provided by the system is similar to the implementation scheme described in the above method, so the specific limitations in one or more visual language-based large model construction site hazard identification system embodiments provided below can refer to the limitations of the visual language-based large model construction site hazard identification method described above, and will not be repeated here.

[0135] In one exemplary embodiment, as shown in Figure 4 a visual language-based large model construction site hazard identification system is provided, comprising:

[0136] The data acquisition module 401 is configured to acquire image data of high-altitude work in a construction site collected in real time.

[0137] The data processing module 402 is configured to pre-process the image data to obtain a to-be-detected image.

[0138] The model algorithm module 403 is configured to input the to-be-detected image into a construction safety visual language model to obtain a hazard identification result, wherein the hazard identification result comprises a high-altitude falling safety hazard type and a standard natural language image caption.

[0139] In one embodiment, the system further comprises:

[0140] The feature extraction module is configured to perform multi-scale language feature extraction on the to-be-detected image to obtain a multi-scale feature vector.

[0141] The hazard identification module is configured to predict a high-altitude falling safety hazard type based on the multi-scale feature vector; and the caption generation module is configured to perform semantic alignment and term mapping on the multi-scale feature vector based on a high-altitude work construction safety standard term database and a term constraint prompt to obtain a standard natural language image caption.

[0142] an output module configured to construct a hazard identification result according to the high-fall safety hazard type and the natural language image caption.

[0143] In one of the embodiments, the method further comprises:

[0144] The data acquisition module 401 is further configured to acquire a target construction safety standard, and extract high-fall hazard-related terms to obtain a plurality of standard terms and corresponding synonyms and near-synonyms thereof;

[0145] The database construction module is configured to construct a high-fall work construction safety specification standard term database using a bidirectional comparison structure according to the standard terms and corresponding synonyms and near-synonyms thereof, in combination with high-fall work normal state standard terms and natural language descriptions thereof corresponding to the safety specification terms.

[0146] In one of the embodiments, the method further comprises:

[0147] The data acquisition module 401 is further configured to acquire image data of high-fall work at a construction site, and perform preprocessing and text labeling to obtain an image-text pair dataset; the image-text pair dataset includes images and corresponding safety hazard types and a plurality of text descriptions;

[0148] The natural language processing module is configured to perform natural language description standard paradigm conversion on the image-text pair dataset to obtain a standardized training dataset;

[0149] The adjustment module is configured to perform structural adjustment on the pre-trained visual language model using a lightweight adapter of an MS-Swift framework and a low-rank matrix, and perform task-oriented multi-task joint training according to the standardized training dataset to obtain a construction safety visual language model.

[0150] In one of the embodiments, the method further comprises:

[0151] The data processing module 402 is further configured to calculate image similarity of the image data using a perceptual hashing algorithm, and save images with image similarity exceeding a preset threshold by one to obtain a deduplicated image set;

[0152] The data processing module 402 is further configured to perform resolution screening on the deduplicated image set, and eliminate low-quality images to obtain a target image set;

[0153] The data processing module 402 is further configured to perform safety hazard labeling on the target image set according to expert experience to obtain safety hazard types and text descriptions corresponding to each image; the safety hazard types include normal conditions, unsafe behaviors of people, and unsafe states of objects;

[0154] The data processing module 402 is further configured to obtain an image-text pair dataset according to each image and the corresponding safety hazard type and text description.

[0155] In one of the embodiments, the method further comprises:

[0156] a key information identification module configured to identify key information of the text description in the image-text pair dataset based on a BERT pre-training language model;

[0157] a data mapping module configured to map the key information to a high-altitude work construction safety specification standard term database, and use a design rule engine to process complex text that cannot be directly mapped to obtain corresponding standard terms;

[0158] a paradigm conversion module configured to replace the corresponding text description in the image-text pair dataset with the standard terms to obtain a standardized training dataset.

[0159] In one of the embodiments, the method further comprises:

[0160] The adjusting module is further configured to fine-tune the target based on minimizing the log-likelihood loss, use a lightweight adapter of an MS-Swift framework and a low-rank matrix to freeze the main body weight of the pre-trained visual language model, optimize the parameters of the image encoder and the text encoder to obtain updated weight matrices;

[0161] a data enhancement module configured to add simulated text noise to the standardized training dataset to obtain enhanced training data;

[0162] a joint training module configured to train the updated weight matrices corresponding to the visual language model based on minimizing a joint loss training target, text prompts and dynamic term constraints, and perform multi-task joint training on the visual language model to obtain model parameters;

[0163] a model construction module configured to construct a construction safety visual language model according to the model parameters.

[0164] In one of the embodiments, the method further comprises:

[0165] a model evaluation module configured to compare the specification natural language image captions generated by the construction safety visual language model with the corresponding text descriptions in the image-text pair dataset through multiple evaluation indexes to obtain evaluation results; the evaluation indexes include word accuracy, semantic alignment degree and syntax structure;

[0166] The adjusting module is further configured to adjust the model parameters of the construction safety visual language model according to the evaluation results.

[0167] The above-described embodiments only express several implementation manners of the application, the description is more specific and detailed, but it cannot be understood as the limitation of the patent scope of the application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the application, several modifications and improvements can be made, which are within the protection scope of the application.

Claims

1. A construction site hidden danger identification method based on a large visual language model, characterized in that: The method comprises: Acquire real-time image data of high-altitude operations at construction sites; Preprocessing the image data to obtain an image to be detected; The image to be detected is input into a construction safety visual language model to obtain a hidden danger identification result; the hidden danger identification result includes the type of height-falling safety hazard and standardized natural language image captions.

2. The method according to claim 1, characterized in that The construction safety visual language model obtains the hidden danger identification result by the following method, including: Performing multi-scale language feature extraction on the image to be detected to obtain a multi-scale feature vector; the multi-scale feature vector includes a person's posture, a scene structure, and an object state; Predicting the type of the height-falling safety hazard based on the multi-scale feature vector; Based on the standard terminology database of construction safety regulations for height operations and terminology constraint prompts, semantic alignment and terminology mapping are performed on the multi-scale feature vectors to obtain the standard natural language image captions; The hazard identification result is obtained according to the type of safety hazard of falling from height and the standard natural language image caption construction.

3. The method according to claim 2, characterized in that The database of standard terminology for construction safety regulations for height operations is constructed by the following method: Obtain target construction safety standards and extract terms related to height-fall hazards, obtaining multiple standard terms and their corresponding synonyms and antonyms; Based on each of the standard terms and their corresponding synonyms and antonyms, combined with the standard terms for normal conditions of height operations corresponding to the safety specification terms and their natural language descriptions, a two-way comparison structure is used to construct the standard terminology database for safety specifications for height operations.

4. The method according to any one of claims 1 to 3, characterized in that The construction safety visual language model is obtained by the following method: Obtain image data of high-altitude operations at a construction site, perform preprocessing and text annotation, and obtain an image-text pair dataset; the image-text pair dataset includes images, their corresponding safety hazard types, and multiple text descriptions; Performing a natural language description standard paradigm conversion on the image-text pair dataset to obtain a standardized training dataset; The pre-trained visual language model is structurally adjusted using a lightweight adapter and a low-rank matrix of the MS-Swift framework, and task-oriented multi-task joint training is performed based on the standardized training dataset to obtain the construction safety visual language model.

5. The method according to claim 4, characterized in that The image data of high-altitude operations at the construction site is obtained, and preprocessed and text-annotated to obtain an image-text pair dataset, including: Calculating the image similarity of the image data using a perceptual hash algorithm, and selecting and saving images whose image similarity exceeds a preset threshold to obtain a deduplicated image set; Performing resolution screening on the deduplicated image set and eliminating low-quality images to obtain a target image set; The target image set is labeled for safety hazards based on expert experience to obtain the safety hazard type and text description corresponding to each image; the safety hazard type includes normal conditions, unsafe human behavior, and unsafe conditions of objects; The image-text pair dataset is obtained according to each of the images and the corresponding safety hazard type and text description.

6. The method according to claim 4, characterized in that The step of performing a natural language description standard paradigm conversion on the image-text pair dataset to obtain a standardized training dataset includes: Recognize key information of the text description in the image-text pair dataset based on the BERT pre-trained language model; Mapping the key information into the standard terminology database of the height operation construction safety specification, and using the design rule engine to process complex text that cannot be directly mapped to obtain corresponding standard terminology; The standard terms are used to replace the corresponding text descriptions in the image-text pair dataset to obtain the standardized training dataset.

7. The method according to claim 4, characterized in that The lightweight adapter and low-rank matrix of the MS-Swift framework are used to structurally adjust the pre-trained visual language model, and task-oriented multi-task joint training is performed based on the standardized training dataset to obtain the construction safety visual language model, including: Based on the minimization of the log-likelihood loss fine-tuning objective, the lightweight adapter of the MS-Swift framework and the low-rank matrix are used to freeze the main weights of the pre-trained visual language model, and the parameters of the image encoder and the text encoder are optimized to obtain an updated weight matrix; Adding simulated text noise to the standardized training data set to obtain enhanced training data; Based on the minimization of the joint loss training objective, text prompts, and dynamic terminology constraints, the visual language model corresponding to the updated weight matrix is ​​subjected to multi-task joint training using the enhanced training data to obtain model parameters; the multi-task corresponds to safety hazard type classification and standardized natural language image caption generation; Constructing the construction safety visual language model according to the model parameters; The log-likelihood loss is obtained by the following formula: Wherein, L is the log-likelihood loss; P(C i |I i ; θ) is given a training data set I i and model parameters θ to generate canonical natural language image captions C i The probability of , including the updated weight matrix W' after adaptation; K is the total number of data samples in the training data set.

8. The method according to claim 7, characterized in that The method of fine-tuning the objective based on minimizing the log-likelihood loss, freezing the main weights of the pre-trained visual language model using the lightweight adapter of the MS-Swift framework and the low-rank matrix, optimizing the parameters of the image encoder and the text encoder, and obtaining an updated weight matrix includes: The updated weight matrix is ​​obtained by the following formula: Among them, W' is the updated weight matrix; W is the main weight matrix, and B is a low-rank matrix, and d and k are the matrix dimensions of W; r is the rank of the lightweight adapter of the MS-Swift framework, and r << min(d, k); α is a scaling parameter used to adjust the impact of the updated weight matrix on the main weight matrix.

9. The method according to claim 4, characterized in that The method further comprises: Comparing the canonical natural language image captions generated by the construction safety visual language model with the corresponding text descriptions in the image-text pair dataset using multiple evaluation indicators to obtain an evaluation result; the evaluation indicators include word accuracy, semantic alignment, and word order structure; The model parameters of the construction safety visual language model are adjusted according to the evaluation results.

10. A construction site hidden danger identification system based on a large visual language model, characterized by: The system comprises: A data acquisition module is used to obtain real-time image data of high-altitude operations at the construction site; A data processing module, used for preprocessing the image data to obtain an image to be detected; The model algorithm module is used to input the image to be detected into the construction safety visual language model to obtain a hidden danger identification result; the hidden danger identification result includes the type of safety hazard of falling from height and standardized natural language image captions.

Images