Equipment installation construction potential safety hazard identification and visual grounding subtitle generation method

Abstract

The invention discloses an equipment installation construction potential safety hazard identification and visual grounding subtitle generation method, and the method comprises the steps: obtaining a to-be-identified construction scene image, and inputting a trained visual grounding image subtitle generation model; according to the method, a visual encoder, a safety semantic projection layer, a multi-mode decoder, an instruction analysis module and a feature fusion module form a visual grounding image subtitle generation model, and the visual grounding image subtitle generation model is further coupled through a construction safety specification database and multi-level labels in the training process. An integrated link from data set construction to visual grounding image subtitle generation model output is formed, so that the visual grounding image subtitle generation model outputs a pixel-level mask corresponding to an image subtitle text description while generating the image subtitle text description containing a potential safety hazard mark; therefore, the result expression of consistent positioning identification and subtitle display is realized, and the interpretability of equipment installation construction potential safety hazard identification output and the field disposal efficiency are improved.

Description

Technical Field

[0001] This invention belongs to the field of equipment installation and construction safety hazard identification technology, specifically relating to a method for identifying equipment installation and construction safety hazards and generating visual grounding subtitles. Background Technology

[0002] With the continuous increase in the scale and complexity of the global construction industry, equipment installation and construction, as a key link in engineering construction, exhibits high frequency, diversity, and interconnected safety hazards. Equipment installation and construction typically involve multiple types of operations such as hoisting, working at heights, temporary power supply, electrical wiring, and equipment commissioning. On-site operations are characterized by complex spatial structures, diverse types of work objects, close connections between work processes, and multiple trades operating concurrently. During construction, the lack of safety protection facilities, abnormal conditions, or non-standard work practices can easily lead to accidents such as mechanical injuries, falls from heights, electric shocks, and falling objects, adversely affecting personnel safety, project progress, and property safety. Therefore, how to achieve timely detection, accurate identification, and effective communication of safety hazards at equipment installation and construction sites is a crucial technical issue in construction safety management. Currently, safety management at equipment installation and construction sites still relies primarily on manual inspections and experience-based judgment. Site managers typically assess work behaviors and the work environment through inspections, manual recording, and post-event summaries. This type of method is difficult to achieve continuous monitoring and comprehensive coverage under conditions such as dispersed construction sites, frequent dynamic changes in work status, and the instantaneous and concealed nature of hidden dangers. At the same time, manual judgment is greatly affected by the professional level and subjective factors of personnel, which can easily lead to missed detections, misjudgments, and inconsistent descriptions. It is difficult to meet the requirements of equipment installation and construction for accurate, timely, and standardized management of safety hazard identification.

[0003] In recent years, with the development of technologies such as computer vision and natural language processing, visual language models based on image understanding and text generation have been gradually introduced into the field of construction scene analysis. These models are used to understand the content of construction images and generate corresponding text descriptions, thus providing a new technical path for the automated expression of information at construction sites. However, existing visual language models are mostly trained on general scene data and lack targeted modeling of typical objects, operational actions, spatial relationships, and safety specification semantics in equipment installation and construction scenarios, resulting in limited understanding and expression capabilities in specialized construction scenarios.

[0004] Specifically, equipment installation and construction images generally feature complex backgrounds, multiple objects coexisting, and densely packed details of operational actions. Safety hazards often manifest in details such as the installation status of protective facilities, the spatial relationship between personnel and equipment / edges, and the standardization of operational procedures. Existing methods, when processing such images, tend to focus on the overall identification of salient objects, paying insufficient attention to construction background elements and key details, leading to omissions of key hazard points or limited coverage of the generated hazard semantic descriptions. Furthermore, the hazard text descriptions output by different models lack unified terminology and expression standards, with inconsistencies in the use of professional terminology and inconsistent descriptive granularity being common, increasing the difficulty of subsequent hazard statistical analysis, standard verification, and management decision-making.

[0005] Meanwhile, in terms of the expression of hazard identification results, existing technologies mainly rely on overall text descriptions or label outputs, which makes it difficult to accurately correspond the semantics of hazards with specific areas in images. They lack intuitive presentation and interpretable support for the location of hazards, which is not conducive to on-site personnel quickly locating hazard risk points and carrying out disposal and rectification. Summary of the Invention

[0006] The purpose of this invention is to address the problems raised in the background art by proposing a method for identifying safety hazards during equipment installation and construction and generating visual grounding subtitles.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] This invention proposes a method for identifying safety hazards during equipment installation and construction and generating visual grounding subtitles, comprising:

[0009] Acquire the construction scene image to be identified and input it into the trained visual grounding image caption generation model;

[0010] The visual grounding image caption generation model includes a visual encoder, a safety semantic projection layer, and a multimodal decoder connected in sequence, as well as an instruction parsing module and a feature fusion module connected in sequence. The construction scene image to be identified is used as the input of the visual encoder, and a preset task instruction prompt is used as the input of the instruction parsing module. The outputs of the visual encoder and the feature fusion module are used as the inputs of the safety semantic projection layer. The output of the safety projection semantic layer is processed by the multimodal decoder to obtain the recognition result of the image caption text description containing safety hazard markers and the regional positioning information of each sentence in the image caption text description in the construction scene image to be identified.

[0011] Preferably, the visual encoder is a CLIP visual encoder, and after the construction scene image to be identified is input into the visual encoder, the construction scene image to be identified is evenly divided into N small blocks. After each small block passes through the visual encoder, a visual block is obtained. The N visual blocks are spliced ​​together to form a visual feature according to the position of each small block before the division.

[0012] Preferably, the instruction parsing module is a Transformer encoding structure, and the task instruction prompts obtain instruction semantic features after passing through the instruction parsing module;

[0013] The feature fusion module employs an attention mechanism. When performing attention mechanism calculations, the instruction semantic features are used as query vectors, and the visual features are used as key and value vectors. After the attention mechanism calculations, a visual context representation is obtained. The visual context representation is then added to the instruction semantic features to obtain the fused feature representation.

[0014] The secure projection semantic layer comprises two multilayer perceptrons connected in sequence. Both the fused feature representation and visual features are used as inputs to the secure projection semantic layer to obtain secure semantic features.

[0015] Preferably, the multimodal decoder includes a text generation submodule and a segmentation mask decoding submodule connected in sequence;

[0016] The task instruction prompts with start tags are segmented to obtain a token sequence, which is then used as input to the text generation submodule.

[0017] The text generation submodule is based on a Transformer decoder and includes a self-attention mechanism, a cross-modal attention mechanism, a feedforward neural network, a linear layer, and a softmax layer connected in sequence. Each of the self-attention mechanism, cross-modal attention mechanism, and feedforward neural network is followed by a residual connection and a layer normalization layer. The token sequence passes through the self-attention mechanism sequentially, and after passing through the corresponding residual connection and layer normalization, a text representation is obtained. In the cross-modal attention mechanism, the text representation serves as the query vector, and visual features and safety semantic features serve as the key vector and value vector, respectively. The output of the text generation submodule is an image caption text description of the construction scene image to be identified. Each statement in the image caption text description describes a region of the construction scene image to be identified. When there is a safety hazard in the construction scene image to be identified, the corresponding statement in the image caption text description is marked with a safety hazard symbol.

[0018] Preferably, the segmentation mask decoding submodule includes N sequentially connected upsampling units, each of which corresponds to one of the N visual blocks in the visual features. Each upsampling unit includes a sequentially connected upsampling layer and a convolutional layer. The image caption text description is used as the input of the segmentation mask decoding submodule. The outputs of the first N-1 upsampling units are concatenated with the corresponding visual blocks and used as the input of the next upsampling unit. The output of the last upsampling unit is concatenated with the corresponding visual block to obtain a mask corresponding to each statement in the image caption text description, which is the output of the segmentation mask decoding submodule. The mask and the image caption text description enable the location and description of each region in the construction scene image to be identified, thereby realizing the location, identification, and caption display of areas with safety hazards in the construction scene image to be identified.

[0019] Preferably, the dataset for the visual grounding image caption generation model is constructed as follows:

[0020] Construct a database of construction safety standards, including:

[0021] Collect safety specification documents related to equipment installation and construction, and use natural language processing technology based on regularized clause parsing and keyword retrieval to segment, identify keywords and parse clauses of the safety specification documents, extract terms related to equipment construction, standardize the expression of each term, and classify each term into safety categories after standardization.

[0022] The standardized terms are mapped one-to-one with the safety clauses in the safety specification documents, establishing a relationship that includes terms, safety categories, content of the specification clauses, specification names, and specification clause numbers. The relationships are then stored in a structured form in the construction safety specification database.

[0023] Preprocessing of the collected construction scene images: Deduplication, removal of low-quality images with resolutions lower than the preset resolution, and size normalization are performed on all collected construction scene images. The preprocessed construction scene images are called standardized construction images.

[0024] Based on the construction safety specification database, each standardized construction image was first annotated with first-level, second-level, third-level, and fourth-level annotations. Then, experts reviewed and corrected the above four-level annotation results. Finally, experts identified the safety hazards in each standardized construction image and supplemented them with corresponding standardized text descriptions to obtain the dataset.

[0025] Preferably, the processing procedure for the first-level annotation is as follows:

[0026] The Tag2Text model and RAM model are used to identify construction objects in various standardized construction images;

[0027] The OV-SAM model and HQ-SAM model were used to segment the identified construction objects, respectively, to obtain ordinary mask labels and high-precision mask labels that correspond one-to-one with each construction object in each standardized construction image.

[0028] The MiDAS model was used to obtain the depth information of each construction object in each standardized construction image.

[0029] The first-level annotation consists of ordinary mask annotations, high-precision mask annotations, and depth information corresponding to each construction object in each standardized construction image.

[0030] Preferably, the processing procedure for the second-level annotation is as follows:

[0031] The BLIP-2 and LLaVA-v1.5 models are used to generate brief scene descriptions for each standardized construction image. Semantic phrases reflecting the interaction between construction objects are extracted from the scene descriptions. The MDETR model is used to match the semantic phrases with the construction objects identified in the corresponding standardized construction images to determine the corresponding positional relationship between the subject and object in the semantic phrases in the standardized construction images, thereby extracting the interaction relationship between each construction object in the standardized construction images.

[0032] Based on the interaction relationships between various construction objects, the work areas in the standardized construction images are divided, and the semantic determination of the work scene type is completed.

[0033] The processing procedure for the third-level annotation is as follows:

[0034] Each construction object in the standardized construction image is used as a node of the scene graph, and the interaction relationship between each construction object is used as the edge of the scene graph to form the scene graph. The scene graph is then used to generate a dense text description containing each construction object and the interaction relationship between each construction object through the Vicuna-v1.5 pre-trained large language model.

[0035] The generated dense text description is verified by combining the nodes and edges of the scene graph to determine whether the dense text description is accurate. If it is inaccurate, the dense text description is regenerated by using the Vicuna-v1.5 pre-trained large language model. This process is repeated until an accurate dense text description is generated.

[0036] Preferably, the processing procedure for the fourth-level annotation is as follows:

[0037] For each relationship in the construction safety specification database, a corresponding text description is generated according to the structure of terminology, safety category, specification clause content, specification name and specification clause number. This text description is then input into the BGE model to obtain a term semantic vector, and all term semantic vectors constitute a term vector library.

[0038] Dense text descriptions are vectorized and encoded, and Top-K nearest neighbor search is performed between the vectorized codes and the term vector library based on cosine similarity. The relationships in the construction safety specification database corresponding to the dense text descriptions are selected, and the terms, safety categories, specification clause content, specification names and specification clause numbers in each relationship are used as the fourth level of annotation for standardized construction images.

[0039] The standardized construction images after first-level, second-level, third-level, and fourth-level annotations are reviewed and corrected by experts. Then, experts identify the safety hazards in the standardized construction images and supplement each annotated standardized construction image with standardized text descriptions related to the safety hazards to obtain the dataset.

[0040] Preferably, the total loss function during the training process of the visual grounding image caption generation model is... for:

[0041] ;

[0042] in,

[0043] ;

[0044] ;

[0045] ;

[0046] in, , and These are text generation loss, mask matching loss, and security semantic loss, respectively. , and These are the weights for text generation loss, mask matching loss, and security semantic loss, respectively. For the first Sample The probability distribution of the corresponding real image caption text description. For the first Sample The probability distribution of the corresponding predicted image caption text description. The DICE loss is used to generate the mask pixel set and the real mask pixel set for the segmentation mask decoding submodule. To generate the IOU loss between the mask pixel set and the real mask pixel set for the segmentation mask decoding submodule, These are the weighting coefficients for the DICE loss. For cosine similarity, The text description for the image captions with safety hazard markers generated by the text generation submodule. This refers to the relevant code clauses in the construction safety code database that relate to safety hazard markings in the image captions and text descriptions. This is the semantic vector obtained after encoding by the BERT model.

[0047] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0048] The method for identifying safety hazards during equipment installation and construction and generating visual grounding captions constructs a visual grounding image caption generation model through a visual encoder, a safety semantic projection layer, a multimodal decoder, an instruction parsing module, and a feature fusion module. During training, the model is further coupled with a construction safety specification database and multi-level annotations, forming an integrated link from dataset construction to the output of the visual grounding image caption generation model. This enables the visual grounding image caption generation model to generate image caption text descriptions containing safety hazard markers while outputting pixel-level masks corresponding to the image caption text descriptions. This achieves consistent results between location recognition and caption display, improving the interpretability of the identified safety hazards during equipment installation and construction and enhancing on-site handling efficiency. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the structure of the visual grounding image subtitle generation model in the equipment installation and construction safety hazard identification and visual grounding subtitle generation method of the present invention;

[0050] Figure 2 A schematic diagram illustrating the process of constructing the dataset for the visual grounding image caption generation model of this invention;

[0051] Figure 3 This is a visual representation of the standardized construction images, images corresponding to ordinary mask annotations, images corresponding to high-precision mask annotations, and images corresponding to depth information of this invention.

[0052] Figure 4 This is an example image of standardized construction images in the dataset of this invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0055] In one embodiment, such as Figures 1-4 As shown, a method for identifying safety hazards during equipment installation and construction and generating visual grounding subtitles is provided, including:

[0056] Step 1: Obtain the construction scene image to be identified and input it into the trained visual grounding image caption generation model;

[0057] Step 2: The visual grounding image caption generation model includes a visual encoder, a safety semantic projection layer, and a multimodal decoder connected in sequence, as well as an instruction parsing module and a feature fusion module connected in sequence. The construction scene image to be identified is used as the input of the visual encoder. A preset task instruction prompt (with a start mark) is set as the input of the instruction parsing module. The outputs of the visual encoder and the feature fusion module are used as the input of the safety semantic projection layer. The output of the safety projection semantic layer is processed by the multimodal decoder to obtain the image caption text description containing safety hazard markers and the recognition result of the area positioning information corresponding to each sentence in the image caption text description in the construction scene image to be identified.

[0058] It should be noted that both the instruction parsing module and the feature fusion module are structures found in the GLaMM model.

[0059] Step 2.1: The visual encoder is a CLIP visual encoder. The construction scene image to be recognized is used as the input to the visual encoder, and the input dimension is ( , , ),in, and These are the height and width of the construction scene image, respectively. Given a 3x3 (i.e., RGB image) grid, the construction scene image to be identified is uniformly divided into N small blocks, each with dimensions ( ). / N, / N, 3). Each small block is processed by a visual encoder to obtain a visual block (each visual block has a feature dimension of ). (e.g., 1024)), the N visual blocks are pieced together according to the positions of the individual blocks before segmentation to form a visual feature (with dimensions (N, )).

[0060] It should be noted that the front network of the CLIP visual encoder is used to represent general visual structural information such as edges, textures, contours and basic geometric shapes in the image; the high-level network of the CLIP visual encoder is used as a high-level feature layer for semantic abstraction and global modeling. Based on the output of the front network, it further performs feature combination and semantic aggregation of the image to characterize the semantic categories and functional attributes of construction objects such as personnel, equipment, components and safety protection equipment in construction images.

[0061] The front-end network of a CLIP visual encoder typically refers to the initial layers responsible for extracting low-level, local visual features. These layers (such as the first few convolutional blocks of ResNet or the first few Transformer layers of ViT) primarily capture basic visual elements of an image, such as edges, textures, and colors, corresponding to local region information in the image. The high-level network of a CLIP visual encoder typically refers to the deep networks responsible for extracting high-level, global semantic features. These layers (such as the later residual blocks of ResNet or the later Transformer layers of ViT) integrate local information to form an abstract representation of the overall image content, object categories, and scene semantics (the concepts and meanings of the front-end and high-level networks of a CLIP visual encoder are disclosed in the existing literature: "PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining").

[0062] Step 2.2: The instruction parsing module uses the Transformer's encoding structure (the specific structure is the existing structure and will not be described in detail). Task instruction prompts are provided (where the task instruction prompts are pre-set, such as "Please describe the scene of the construction scene image," and the dimension of the task instruction prompt is (T, ...). ), where T is the length (number of words) of the task instruction prompt. After the input word vectors (with dimensions of (1, 2, 3) are processed by the instruction parsing module, the instruction semantic features are obtained (high-dimensional semantic feature vectors representing the task instructions, with dimensions of (1, 2, 3)). ), (This refers to the dimension of instruction semantic features).

[0063] It should be noted that the instruction semantic features generated by the instruction parsing module are input to the feature fusion module as language-side semantic constraint signals. This is used to guide the visual grounding image caption generation model to focus on construction objects, work behaviors, and safety semantic elements related to the current task instruction prompts during the output process.

[0064] Step 2.3: The feature fusion module employs an attention mechanism. During attention mechanism calculation, the instruction semantic features are used as the query vector, and the visual features are used as the key and value vectors. The attention mechanism then yields a visual context representation (with dimensions (N, ...). The visual context representation is then added to the instruction semantic features to obtain a fused feature representation (with dimensions (N, ...). ) or (1, ));

[0065] It should be noted that the feature fusion module is used to achieve cross-modal alignment and joint modeling between visual features and instruction semantic features. The feature fusion module takes the visual features output by the visual encoder and the instruction semantic features output by the instruction parsing module as inputs, and performs correlation modeling of different modal features through a cross-modal attention mechanism, so that the language side semantics can selectively focus on relevant regions in the image, while mapping visual information to a unified multimodal semantic space.

[0066] Step 2.4: The secure projection semantic layer consists of two sequentially connected multilayer perceptrons. Both the fused feature representation and visual features are used as inputs to the secure projection semantic layer to obtain secure semantic features (with dimensions (N, ...). )).

[0067] The first layer, a multilayer perceptron, is used for dimension alignment and feature transformation. It includes at least one fully connected layer and a non-linear activation function (such as the QuickGELU activation function) to transform the input features from dimensionality alignment to feature transformation. Mapping to intermediate dimensions The second multilayer perceptron is used for semantic mapping and discrimination enhancement. It includes at least one fully connected layer to transform intermediate features from dimensionality... Mapped to a dimension consistent with the embedding space of security specification terminology In a specific embodiment, , , The first multilayer perceptron has a two-layer fully connected structure, and the second multilayer perceptron has a one-layer fully connected structure. The first multilayer perceptron includes two fully connected layers ( The second layer of the multilayer perceptron includes a fully connected layer ( ),in and This is a preset intermediate dimension used to achieve a balance between parameter size and expressive power; however, the present invention is not limited to the above-mentioned layer and neuron number settings, and those skilled in the art can make equivalent adjustments according to the encoder output dimension and task requirements.

[0068] It should be noted that the first layer of the multilayer perceptron primarily undertakes the functions of feature transformation and dimensional alignment. It performs linear mapping and nonlinear activation processing on the fused feature representation and visual features, respectively, projecting both types of features onto intermediate features consistent with the scale of the term embedding space. This reduces redundant information unrelated to safety semantics and achieves semantic alignment of features from different sources. Subsequently, the two intermediate features are combined through feature-level fusion to form a unified intermediate semantic feature representation. Building upon this, the second layer of the multilayer perceptron further performs semantic mapping and discriminative enhancement processing on the intermediate semantic feature representation, mapping it to the professional semantic space defined by equipment installation and construction terms. This makes the final output safety semantic features more closely resemble the semantic structure of the terms in terms of distribution, thereby improving the model's expressive and discriminative capabilities for construction safety semantics.

[0069] Step 2.5: The multimodal decoder includes a text generation submodule and a segmentation mask decoding submodule connected in sequence;

[0070] Step 2.5.1: Segment the task instruction prompts with start tags to obtain a token sequence (dimension (T, ...). The token sequence serves as the input to the text generation submodule.

[0071] The text generation submodule is built on a Transformer decoder and includes a self-attention mechanism, a cross-modal attention mechanism, a feedforward neural network, a linear layer, and a softmax layer connected in sequence. Each of the self-attention mechanism, cross-modal attention mechanism, and feedforward neural network is followed by a residual connection and a layer normalization layer. The token sequence passes through the self-attention mechanism sequentially, and after passing through the corresponding residual connection and layer normalization, a text representation is obtained. In the cross-modal attention mechanism, the text representation serves as the query vector, while visual features and safety semantic features serve as cross-modal context features, jointly participating in the attention calculation to construct key and value vectors (where visual features provide the appearance and spatial information of the construction object, and safety semantic features guide the model to focus on construction object attributes and construction operation semantics related to safety regulations during the generation process). The output of the text generation submodule is an image caption text description of the construction scene image to be identified, and each element in the image caption text description... The text descriptions are for different regions of the construction scene image to be identified. When there are safety hazards in the construction scene image, the corresponding text descriptions are marked with safety hazards. (The generated text descriptions provide an overall description of the construction objects, work activities, and their spatial relationships in the construction scene image. When a potential safety hazard is involved in the construction scene image, the text generation submodule automatically matches the relevant standard clauses in the construction safety standard database based on the type and location of the safety hazard, and generates standardized text descriptions that include safety hazard markers. In normal construction scene images without safety hazards, the text generation submodule still retrieves relevant terms from the safety standard database based on the objects, work activities, and their spatial relationships in the construction scene image, and generates standardized text descriptions. This ensures that the content of the generated text descriptions conforms to standardization requirements and that the expression method conforms to the standards in the construction safety standard database.)

[0072] It should be noted that when the text generation submodule predicts a construction object or construction operation status related to a safety hazard, it inserts a safety hazard marker at the corresponding text location based on the correspondence between visual features and safety semantic features learned by the model during training (such as safety specifications annotated at level four and the ground truth values ​​of safety hazards confirmed by experts). This ensures that the generated dense text description clearly distinguishes between "ordinary object descriptions" and "hazard-related descriptions" at the linguistic level. This safety hazard marker is not additional input information but is automatically generated by the text generation submodule under the condition of integrating visual features and safety semantic features. Its location and quantity reflect the model's judgment result on potential safety hazards.

[0073] Step 2.5.2: The segmentation mask decoding submodule includes N sequentially connected upsampling units. Each of the N upsampling units corresponds one-to-one with one of the N visual blocks in the visual features. Each upsampling unit includes a sequentially connected upsampling layer and a convolutional layer. The image caption text description is used as the input to the segmentation mask decoding submodule. The outputs of the first N-1 upsampling units are concatenated with the corresponding visual blocks and used as the input to the next upsampling unit. The output of the last upsampling unit is concatenated with the corresponding visual block to obtain the mask corresponding to each statement in the image caption text description, which is the output of the segmentation mask decoding submodule (dimension is ( , The mask and image caption text description realize the positioning and description of each area in the construction scene image to be identified, thereby realizing the positioning and identification of areas with safety hazards in the construction scene image to be identified and the caption display (completing the positioning and visualization output of areas with safety hazards).

[0074] It should be noted that the segmentation mask decoding submodule outputs the image region location result corresponding to the image caption semantics (i.e., the image caption text description) through joint analysis of the image caption text description and visual features. It also provides the spatial location of key objects or safety hazard elements in the construction scene image in the form of pixel-level segmentation masks or region markers, realizing the accurate correspondence between the image caption text description and the image spatial region. When the image caption text description contains safety hazard markers, the location and recognition of the safety hazard region in the construction scene image and the display of the image captions are realized based on the location of the safety hazard region and the statements of the safety hazard markers in the image caption text description.

[0075] In another embodiment, such as Figure 2 As shown, the dataset for the visual grounding image caption generation model is constructed as follows during the training process:

[0076] Step I: Construct a construction safety code database, including:

[0077] Step I.1: Collect safety regulations and documents related to equipment installation and construction (collect national standards, industry standards, and technical specifications related to equipment installation and construction, including "General Specifications for Construction and Acceptance of Mechanical Equipment Installation Engineering GB50231-2009", "Technical Specifications for Temporary Power Supply Safety at Construction Sites JGJ46-2005", "Technical Specifications for Safety of High-Altitude Operations in Building Construction JGJ 80-2016", and "Technical Specifications for Safety of Lifting and Hoisting Engineering in Building Construction JGJ 80-2016"). Safety specification documents, such as "276-2012", are obtained in electronic text or structured document form and serve as a source of safety specification knowledge. Natural language processing techniques based on regularized clause parsing and keyword retrieval are used to segment, identify keywords, and parse clauses in these documents. Specifically, the `pdfminer.six` module (or an equivalent PDF text parsing tool) in Python is used to extract safety specification documents for equipment installation and construction. The `re` module in Python is used to perform regularized parsing and clause-level segmentation of the specification clause numbering patterns (such as "3.0.5", "8.1", "10.2.3"), combined with Chinese word segmentation tools (such as `jieba`) and Python's built-in string matching functions / methods (such as `in`, `str.find()`, and `str.index()`). (e.g., segmenting, identifying keywords, and parsing clauses in safety specifications documents for equipment installation and construction), extracting terms related to equipment construction (including at least terms related to construction objects, work actions, protective facilities, and safety status), standardizing the expression of each term, and supplementing the safety category to which the term belongs (e.g., "electrical safety" and "equipment safety") to ensure the accuracy and consistency of the terminology, standardizing the expression of each term, and classifying each standardized term into a safety category (e.g., safety categories such as safety for working at heights, hoisting and lifting, electrical safety, and equipment operation safety).

[0078] Specifically, the standardization of terminology involves: using rule-driven synonym normalization and semantic constraint disambiguation strategies to standardize safety terminology, including unifying the naming of synonymous terms (such as "lifting machinery" and "crane") and clarifying the semantics of ambiguous terms (such as "high-altitude operations") (e.g., clearly defining the operating height range of high-altitude operations).

[0079] Step 1.2: Map each standardized term to the safety specification clauses in the safety specification documents, establishing a relationship between the term, safety category, specification clause content (i.e., the specific content of the specification clause), specification name, and specification clause number. Store these relationships in a structured format in the construction safety specification database. An example of the construction safety specification database is shown in Table 1.

[0080] Table 1

[0081]

[0082] Among them, the construction safety specification database supports querying and calling by term name, safety category or specification clause number, providing safety specification semantic support for subsequent multi-level annotation of construction scene images and visual grounding image caption generation model.

[0083] Step II: Preprocess the collected construction scene images: Deduplication of all collected construction scene images is performed (using the perceptual hash algorithm (pHash) to calculate the similarity of the collected construction scene images, removing duplicate or highly similar construction scene images with a similarity higher than 95%), removal of low-quality images with a resolution lower than the preset resolution (removing low-quality images with a resolution lower than 1024×768, and excluding images with obvious overexposure or severe blurring), and size normalization processing (size normalization performs proportional scaling processing on ultra-large images with a resolution higher than 4096×4096, removes images with a resolution lower than 512×512, and retains construction scene images with a resolution that meets the model input requirements). The preprocessed construction scene images are called standardized construction images.

[0084] Step III: Based on the construction safety specification database, firstly, each standardized construction image is annotated sequentially at the first, second, third, and fourth levels. Secondly, experts review and correct the results of these four levels of annotation. Finally, experts identify safety hazards in each standardized construction image and supplement them with corresponding standardized text descriptions, resulting in a dataset including:

[0085] Step III.1, the processing procedure for the first-level annotation is as follows:

[0086] The Tag2Text model and RAM model are used to identify construction objects in standardized construction images (where construction objects include: personnel, equipment, components, protective facilities, and tools). The Tag2Text model is used to generate overall semantic labels related to the construction scene, while the RAM model focuses on the perception and differentiation of different types of construction objects, thereby obtaining the category information of construction objects and their basic attributes such as appearance features and protection status.

[0087] The identified construction objects were segmented using the OV-SAM model and the HQ-SAM model, respectively. The OV-SAM model was used to generate a general object mask with relatively complete coverage to achieve preliminary regional localization of the construction objects. Based on this, the HQ-SAM model refined the target boundaries to obtain higher-precision segmentation results, thereby improving the quality of the construction object mask in complex construction scenarios. Ordinary mask annotations and high-precision mask annotations were obtained that correspond one-to-one with each construction object in each standardized construction image.

[0088] The MiDAS model is used to obtain the depth information of each construction object in each standardized construction image (to clarify the relative position of the specific object in the scene, such as "the worker is on a 3-meter-high operating platform" and "the crane boom is 10 meters above the ground", laying the foundation for subsequent analysis of inter-object relationships and scene understanding).

[0089] The first-level annotation results are composed of ordinary mask annotations, high-precision mask annotations, and depth information corresponding to each construction object in each standardized construction image.

[0090] in, Figure 3 The text displays a visual representation of standardized construction images, images corresponding to ordinary mask annotations of construction objects within standardized construction images, images corresponding to high-precision mask annotations of construction objects within standardized construction images, and images corresponding to depth information of construction objects within standardized construction images. Figure 3 In the image, (a) is the standardized construction image, (b) is the image corresponding to the depth information (i.e., the depth map), (c) is the image corresponding to the ordinary mask annotation, and (d) is the image corresponding to the high-precision mask annotation.

[0091] The above-mentioned Tag2Text, RAM, OV-SAM, HQ-SAM, and MiDAS models are all pre-trained models that are directly applied to data annotation, without involving additional training or parameter fine-tuning for the data of this invention.

[0092] Step III.2, the processing procedure for the second-level annotation is as follows:

[0093] The BLIP-2 and LLaVA-v1.5 models were used to generate brief scene descriptions for each standardized construction image. (The BLIP-2 model focuses on generating relatively objective and general scene descriptions from the overall visual content to depict the main objects and their basic states in the image. Based on this, the LLaVA-v1.5 model further provides more expressive scene descriptions, which helps to supplement the action and interaction information between objects.) Semantic phrases reflecting the interaction relationships between construction objects (such as "personnel standing on scaffolding" and "crane equipment hoisting components") were extracted from the scene descriptions. The MDETR model was then used to match the semantic phrases with the construction objects identified in the corresponding standardized construction images to determine the corresponding positional relationship between the subject and object in the semantic phrases in the standardized construction images. This allowed for the extraction of the interaction relationships between various construction objects in the standardized construction images (such as the interaction relationships between personnel and equipment, personnel and work areas, and equipment and components).

[0094] Based on the interaction relationships between various construction objects, the work areas in the standardized construction images are divided, and the semantic determination of the work scene type is completed. Specifically, based on the key construction object combinations (such as lifting equipment, scaffolding, operating platforms, component stacking areas, etc.), the spatial relationship characteristics between construction objects (such as height relationship, support relationship, hoisting relationship), and the interaction relationships described by semantic phrases in the standardized construction images, the work scenes corresponding to the standardized construction images are classified and determined in accordance with the definition rules of different work scenes in the construction safety specification database. These include "lifting and heavy equipment operation area", "high-altitude operation platform", "material stacking and handling area", "factory equipment installation area", etc., so that the type and environmental characteristics of the equipment installation construction scene in the standardized construction images can be clearly defined.

[0095] The BLIP-2, LLaVA-v1.5, and MDETR models mentioned above are all pre-trained models that are directly applied to data annotation, without involving additional training or parameter fine-tuning for the data of this invention.

[0096] Step III.3, the processing procedure for third-level annotations is as follows:

[0097] Each construction object in the standardized construction image is used as a node of the scene graph, and the interaction relationship between each construction object is used as the edge of the scene graph to form the scene graph. The scene graph is then used to generate a dense text description containing each construction object and the interaction relationship between each construction object through the Vicuna-v1.5 pre-trained large language model.

[0098] The generated dense text description is validated by combining the nodes and edges of the scene graph to determine its accuracy. If inaccurate, the dense text description is regenerated using the Vicuna-v1.5 pre-trained large language model. This process is repeated until an accurate dense text description is generated (the dense text description covers a joint expression of the construction object and the work behavior; specifically, the prompts in the Vicuna-v1.5 pre-trained large language model require that, while generating the dense text description, the candidate facts on which it is based be explicitly listed (e.g., a list of construction objects and relationships in the form of triples or "subject-relation-object"), thus obtaining a set of candidate facts that can be parsed by the program or rules); subsequently, the dense text description is validated by comparing the set of candidate facts with the set of nodes and edges in the scene graph item by item, checking the construction objects mentioned in the dense text description. The system checks whether the object exists in the node set, whether the mentioned relation exists in the edge set, and whether the direction of the relation matches the participating construction objects. If the dense text description contains construction objects or relations that do not exist in the scene graph, or if the direction of the relation does not match the participating construction objects, it is judged as failing. If it is judged as failing, the dense text description is regenerated under the same scene graph constraints, and then the dense text description is verified. The above steps are repeated until the construction objects and relations contained in the dense text description are supported in the scene graph (i.e., the generated dense text description passes the verification). Through the above verification and regeneration mechanism, "illusionary supplementation" can be suppressed without changing the parameters of the Vicuna-v1.5 pre-trained large language model, so that the final generated dense text description more stably covers the semantic information such as construction behavior and operation status in the standardized construction image.

[0099] The Vicuna-v1.5 model is directly applied to data annotation as a pre-trained model, without involving additional training or parameter fine-tuning for the data of this invention.

[0100] Step III.4, the processing procedure for the fourth-level annotation is as follows:

[0101] For each relationship in the construction safety specification database, a corresponding text description is generated according to the structure of terminology, safety category, specification clause content, specification name and specification clause number. This text description is then input into the BGE model to obtain a term semantic vector, and all term semantic vectors constitute a term vector library.

[0102] Dense text descriptions are vectorized and encoded. Based on cosine similarity, the vectorized codes are searched against the term vector database using Top-K nearest neighbor retrieval. Valid matches are then selected by combining similarity thresholds. This process identifies the relationships in the construction safety specification database corresponding to the dense text descriptions. The terms, safety categories, specification clauses, specification names, and specification clause numbers in each relationship are used as the fourth-level annotations for standardized construction images.

[0103] The standardized construction images after first-level, second-level, third-level, and fourth-level annotations undergo expert review and correction. Experts then identify potential safety hazards within these images and supplement each annotated image with standardized text descriptions related to these hazards. This results in a dataset (i.e., experts with experience in equipment installation, construction, and safety management review and correct the multi-level annotations of the standardized construction images, determine the presence and types of safety hazards, and supplement each image with standardized text descriptions related to these hazards). The standardized construction images and their standardized text descriptions (including descriptions of normal conditions and those with safety hazards) obtained after this step constitute the equipment installation and construction safety hazard dataset for subsequent training and evaluation of the visual grounding image caption generation model (samples of this dataset are shown below). Figure 4 As shown):

[0104] in Figure 4 The example shown is a standardized construction image from the dataset, and each standardized construction image corresponds to a standardized text description related to safety hazards formed by multiple experts (there are five different experts in this example). One standardized construction image and its corresponding standardized text description are used as a sample.

[0105] Figure 4 (e) in the text represents a standardized construction image under normal circumstances (i.e., without safety hazards), and the standardized text formed by five different experts describes it as follows: the workers wear protective measures in accordance with regulations during the cutting operation; the operators are properly protected when cutting the sheet metal; the workers have complete protective equipment during the cutting operation; the workers receive sufficient safety protection during the cutting operation; and the operators have adequate personal protective measures during the cutting operation. Figure 4 (f) in the text represents the standardized construction image corresponding to the safety hazards that occurred during the operation process. The standardized text description formed by five different experts is as follows: there is residual oxide slag in the cutting area, which was not cleaned according to the specifications; the oxide slag at the work site was not removed according to the standards; the oxide slag in the cutting work area was not cleaned up in accordance with regulations; the operators did not remove the oxide slag generated during cutting as required; there is residual oxide slag in the cutting work area, which lacks standardized treatment. Figure 4(g) in the figure represents the standardized construction image corresponding to the safety hazard of tool defects, and the standardized text description formed by five different experts is as follows: the compressed air pipeline is missing a connection joint; the pneumatic pipeline is not equipped with the required connection parts; the air pressure pipeline is missing the necessary connection parts; the compressed air pipeline is not installed with the specified pipe joint; and the connection interface in the air source pipeline is missing. Figure 4 (h) in the text represents the standardized construction image corresponding to the safety hazard of the equipment defect, and the standardized text description formed by five different experts is as follows: the pressure plate of the manual hoist is missing; the pressure plate buckle is not installed on the manual hoist in use; the fixed pressure plate on the manual hoist is not configured; the clamping pressure plate is missing on the manual hoist used for operation; the chain hoist currently in use lacks a fastening pressure plate. Figure 4 (i) in the figure represents the standardized construction image corresponding to the electrical safety hazard, and the standardized text description formed by five different experts is as follows: the distribution cabinet door is open; the electrical box door is not closed; the distribution panel door is not closed as required; the power cabinet door is not closed in accordance with regulations; and the electrical control cabinet door is left open. Figure 4 (j) in the figure represents the standardized construction image corresponding to the safety hazards of hoisting operations, and the standardized text description formed by five different experts is as follows: During the hoisting of steel, the ground personnel's position is unsafe; during the material hoisting operation, the standing position of the operators is risky; during the hoisting operation, the placement of steel on the ground and the position of personnel are unreasonable; during the steel transfer process, there are safety hazards in the position of ground personnel; during the high-altitude hoisting operation, there are safety issues in the position of personnel. Figure 4 In the image (k), there is a standardized construction image corresponding to the safety hazard of working at height. The standardized text description formed by five different experts is as follows: the worker is sitting on the edge of the platform railing; the employee is leaning against the edge of the platform railing; the operator is working at the edge of the platform railing; the worker is sitting on the top edge of the platform railing; the worker is staying at the edge of the platform railing. Figure 4 (l) in the text represents the standardized construction image corresponding to the safety hazard of power outage, and the standardized text description formed by five different experts is as follows: the power outage sign on the control cabinet is not sealed as required; the power outage sign on the control panel lacks the necessary seal; the power outage sign is not completely sealed and fixed at the control cabinet; the power outage sign on the control cabinet is not sealed in a standardized manner; the power outage sign on the control panel is not sealed in accordance with safety regulations. Figure 4 (m) represents the standardized construction image corresponding to the safety hazards of energy conservation and environmental protection, and the standardized text description formed by five different experts is as follows: the external lighting of the factory building was not turned off in time; the external lights of the factory building were not turned off on time; the outdoor lighting of the factory area was not turned off at the specified time; the lighting facilities outside the factory building were not turned off in time; the external lighting of the factory was not turned off as required.

[0106] The dataset is divided into a 70% training set, a 15% validation set, and a 15% test set; the parameter values ​​are set as follows during training:

[0107] Number of GPUs: 1; Number of training epochs: 4; Number of steps per epoch: 4500; Micro-batch size per GPU: 2; Gradient accumulation steps: 10; Batch size: 20; Optimizer: AdamW; Learning rate: Weight decay: 0.0; Betas: (0.9, 0.95); Gradient clipping: 1.0; Optimization strategy: ZeRO Stage 2; Scheduler type: WarmupDecayLR; Minimum warm-up learning rate: 0; Maximum warm-up learning rate: Preheating steps: 100; Precision setting: BF16; Mixed precision strategy: DeepSpeed ​​BF16.

[0108] The total loss function during the training process of the visual grounding image caption generation model for:

[0109] ;

[0110] in,

[0111] ;

[0112] ;

[0113] ;

[0114] ;

[0115] ;

[0116] in, , and These are text generation loss, mask matching loss, and security semantic loss, respectively. , and These are the weights for text generation loss, mask matching loss, and security semantic loss, respectively. For the first Sample The probability distribution of the corresponding real image caption text description. For the first Sample The probability distribution of the corresponding predicted image caption text description. Generate a set of mask pixels for the segmentation mask decoding submodule. With the set of real mask pixels DICE losses, Generate a set of mask pixels for the segmentation mask decoding submodule. With the set of real mask pixels IOU loss, This is the weighting coefficient for the DICE loss (e.g., 0.5). For cosine similarity, The text description for the image captions with safety hazard markers generated by the text generation submodule. This refers to the relevant code clauses in the construction safety code database that relate to safety hazard markings in the image captions and text descriptions. This is the semantic vector obtained after encoding by the BERT model.

[0117] It should be noted that the text generation loss uses a cross-entropy loss function to ensure that the generated subtitles are grammatically correct and conform to the expression habits of equipment installation and construction scenarios; the cross-entropy loss function measures the difference between the probability distribution of the model-generated subtitles and the probability distribution of the real subtitles. The mask matching loss combines DICE loss and IoU loss to ensure accurate alignment between the segmentation mask and the object described in the text; the IoU loss measures the ratio of the intersection and union between the model-generated mask and the real mask, while the DICE loss evaluates the matching degree between the model-generated mask and the real mask from another perspective. The safety semantic loss uses the BERT model to calculate the semantic similarity between the model-generated subtitle text containing safety hazard descriptions and the content of the regulatory clauses, ensuring the professionalism of the model's generated safety hazard descriptions. The BERT model is pre-trained on large-scale text data and possesses powerful text semantic understanding capabilities. The regulatory clauses are sourced from a construction safety specification database. The model-generated subtitle text containing safety hazard descriptions and the regulatory clauses are input into the BERT model together, and their semantic similarity is calculated.

[0118] The parameters of the visual grounding image caption generation model are updated by backpropagation based on the total loss function until the visual grounding image caption generation model converges, thus obtaining the trained visual grounding image caption generation model.

[0119] During training, the network weights of the front-end network of the visual encoder are frozen to maintain its stable perception of general visual patterns. Simultaneously, the parameters of the higher-level networks are fine-tuned to enable the visual encoder to better adapt to the domain characteristics of equipment installation and construction scenarios within the GLaMM framework. This results in the output of visual feature representations that combine general visual representation capabilities with construction scenario discrimination capabilities, providing high-quality visual input for subsequent visual-language alignment, relational reasoning, and safety semantic generation. The parameters of the instruction parsing module and the feature fusion module are also frozen.

[0120] For the safety projection semantic layer, the relationships and corresponding visual features in the construction safety specification database are used as supervision signals to optimize the parameters of the safety semantic projection layer end-to-end. Through the hierarchical mapping of two cascaded MLP layers, the model can learn the correspondence between the visual features of construction objects and their relationships, thereby stably projecting general visual semantics into the professional semantic space of equipment installation and construction safety, and mitigating the semantic bias problem of the general model in understanding professional terms.

[0121] The method for identifying safety hazards during equipment installation and construction and generating visual grounding captions constructs a visual grounding image caption generation model through a visual encoder, a safety semantic projection layer, a multimodal decoder, an instruction parsing module, and a feature fusion module. During training, the model is further coupled with a construction safety specification database and multi-level annotations, forming an integrated link from dataset construction to the output of the visual grounding image caption generation model. This enables the visual grounding image caption generation model to generate image caption text descriptions containing safety hazard markers while outputting pixel-level masks corresponding to the image caption text descriptions. This achieves consistent results between location recognition and caption display, improving the interpretability of the identified safety hazards during equipment installation and construction and enhancing on-site handling efficiency.

[0122] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for identifying safety hazards during equipment installation and construction and generating visual grounding subtitles, characterized in that: The method for identifying safety hazards during equipment installation and construction and generating visual grounding subtitles includes: Acquire the construction scene image to be identified and input it into the trained visual grounding image caption generation model; The visual grounding image caption generation model includes a visual encoder, a safety semantic projection layer, and a multimodal decoder connected in sequence, as well as an instruction parsing module and a feature fusion module connected in sequence. The construction scene image to be identified is used as the input of the visual encoder, and a preset task instruction prompt is used as the input of the instruction parsing module. The outputs of the visual encoder and the feature fusion module are used as the inputs of the safety semantic projection layer. The output of the safety projection semantic layer is processed by the multimodal decoder to obtain the recognition result of the image caption text description containing safety hazard markers and the regional positioning information of each sentence in the image caption text description in the construction scene image to be identified.

2. The method for identifying safety hazards during equipment installation and construction and generating visual grounding subtitles as described in claim 1, characterized in that: The visual encoder is a CLIP visual encoder. After the construction scene image to be identified is input into the visual encoder, the construction scene image to be identified is evenly divided into N small blocks. After each small block passes through the visual encoder, a visual block is obtained. The N visual blocks are spliced ​​together to form a visual feature according to the position of each small block before the division.

3. The method for identifying safety hazards during equipment installation and construction and generating visual grounding subtitles as described in claim 2, characterized in that: The instruction parsing module uses the Transformer encoding structure. After the task instruction prompt passes through the instruction parsing module, the instruction semantic features are obtained. The feature fusion module employs an attention mechanism. When performing attention mechanism calculations, the instruction semantic features are used as query vectors, and the visual features are used as key and value vectors. After the attention mechanism calculations, a visual context representation is obtained. The visual context representation is then added to the instruction semantic features to obtain the fused feature representation. The secure projection semantic layer comprises two multilayer perceptrons connected in sequence. Both the fused feature representation and visual features are used as inputs to the secure projection semantic layer to obtain secure semantic features.

4. The method for identifying safety hazards during equipment installation and construction and generating visual grounding subtitles as described in claim 3, characterized in that: The multimodal decoder includes a text generation submodule and a segmentation mask decoding submodule connected in sequence; The task instruction prompts with start tags are segmented to obtain a token sequence, which is then used as input to the text generation submodule. The text generation submodule is based on a Transformer decoder and includes a self-attention mechanism, a cross-modal attention mechanism, a feedforward neural network, a linear layer, and a softmax layer connected in sequence. Each of the self-attention mechanism, cross-modal attention mechanism, and feedforward neural network is followed by a residual connection and a layer normalization layer. The token sequence passes through the self-attention mechanism sequentially, and after passing through the corresponding residual connection and layer normalization, a text representation is obtained. In the cross-modal attention mechanism, the text representation serves as the query vector, and visual features and safety semantic features serve as the key vector and value vector, respectively. The output of the text generation submodule is an image caption text description of the construction scene image to be identified. Each statement in the image caption text description describes a region of the construction scene image to be identified. When there is a safety hazard in the construction scene image to be identified, the corresponding statement in the image caption text description is marked with a safety hazard symbol.

5. The method for identifying safety hazards during equipment installation and construction and generating visual grounding subtitles as described in claim 4, characterized in that: The segmentation mask decoding submodule includes N sequentially connected upsampling units, each corresponding to one of the N visual blocks in the visual features. Each upsampling unit includes a sequentially connected upsampling layer and a convolutional layer. The image caption text description is used as the input of the segmentation mask decoding submodule. The outputs of the first N-1 upsampling units are concatenated with the corresponding visual blocks and used as the input of the next upsampling unit. The output of the last upsampling unit is concatenated with the corresponding visual block to obtain the mask corresponding to each statement in the image caption text description, which is the output of the segmentation mask decoding submodule. The mask and the image caption text description enable the localization and description of each region in the construction scene image to be identified, thereby realizing the localization and identification of areas with safety hazards in the construction scene image to be identified and the display of captions.

6. The method for identifying safety hazards during equipment installation and construction and generating visual grounding subtitles as described in claim 1, characterized in that: The dataset for the visual grounding image caption generation model is constructed as follows: Construct a database of construction safety standards, including: Collect safety specification documents related to equipment installation and construction, and use natural language processing technology based on regularized clause parsing and keyword retrieval to segment, identify keywords and parse clauses of the safety specification documents, extract terms related to equipment construction, standardize the expression of each term, and classify each term into safety categories after standardization. The standardized terms are mapped one-to-one with the safety clauses in the safety specification documents, establishing a relationship that includes terms, safety categories, content of the specification clauses, specification names, and specification clause numbers. The relationships are then stored in a structured form in the construction safety specification database. Preprocessing of the collected construction scene images: Deduplication, removal of low-quality images with resolutions lower than the preset resolution, and size normalization are performed on all collected construction scene images. The preprocessed construction scene images are called standardized construction images. Based on the construction safety specification database, each standardized construction image was first annotated with first-level, second-level, third-level, and fourth-level annotations. Then, experts reviewed and corrected the above four-level annotation results. Finally, experts identified the safety hazards in each standardized construction image and supplemented them with corresponding standardized text descriptions to obtain the dataset.

7. The method for identifying safety hazards during equipment installation and construction and generating visual grounding subtitles as described in claim 6, characterized in that: The processing procedure for the first-level annotation is as follows: The Tag2Text model and RAM model were used to identify construction objects in various standardized construction images; The OV-SAM model and HQ-SAM model were used to segment the identified construction objects, respectively, to obtain ordinary mask labels and high-precision mask labels that correspond one-to-one with each construction object in each standardized construction image. The MiDAS model was used to obtain the depth information of each construction object in each standardized construction image. The first-level annotation consists of ordinary mask annotations, high-precision mask annotations, and depth information corresponding to each construction object in each standardized construction image.

8. The method for identifying safety hazards during equipment installation and construction and generating visual grounding subtitles as described in claim 6, characterized in that: The processing procedure for the second-level annotation is as follows: The BLIP-2 and LLaVA-v1.5 models are used to generate brief scene descriptions for each standardized construction image. Semantic phrases reflecting the interaction between construction objects are extracted from the scene descriptions. The MDETR model is used to match the semantic phrases with the construction objects identified in the corresponding standardized construction images to determine the corresponding positional relationship between the subject and object in the semantic phrases in the standardized construction images, thereby extracting the interaction relationship between each construction object in the standardized construction images. Based on the interaction relationships between various construction objects, the work areas in the standardized construction images are divided, and the semantic determination of the work scene type is completed. The processing procedure for the third-level annotation is as follows: Each construction object in the standardized construction image is used as a node of the scene graph, and the interaction relationship between each construction object is used as the edge of the scene graph to form the scene graph. The scene graph is then used to generate a dense text description containing each construction object and the interaction relationship between each construction object through the Vicuna-v1.5 pre-trained large language model. The generated dense text description is verified by combining the nodes and edges of the scene graph to determine whether the dense text description is accurate. If it is inaccurate, the dense text description is regenerated by using the Vicuna-v1.5 pre-trained large language model. This process is repeated until an accurate dense text description is generated.

9. The method for identifying safety hazards during equipment installation and construction and generating visual grounding subtitles as described in claim 6, characterized in that: The processing procedure for the fourth-level annotation is as follows: For each relationship in the construction safety specification database, a corresponding text description is generated according to the structure of terminology, safety category, specification clause content, specification name and specification clause number. This text description is then input into the BGE model to obtain a term semantic vector, and all term semantic vectors constitute a term vector library. Dense text descriptions are vectorized and encoded, and Top-K nearest neighbor search is performed between the vectorized codes and the term vector library based on cosine similarity. The relationships in the construction safety specification database corresponding to the dense text descriptions are selected, and the terms, safety categories, specification clause content, specification names and specification clause numbers in each relationship are used as the fourth level of annotation for standardized construction images. The standardized construction images after first-level, second-level, third-level, and fourth-level annotations are reviewed and corrected by experts. Then, experts identify the safety hazards in the standardized construction images and supplement each annotated standardized construction image with standardized text descriptions related to the safety hazards to obtain the dataset.

10. The method for identifying safety hazards during equipment installation and construction and generating visual grounding subtitles as described in claim 5, characterized in that: The total loss function during the training process of the visual grounding image caption generation model for: ； in, ； ； ； in, , and These are text generation loss, mask matching loss, and security semantic loss, respectively. , and These are the weights for text generation loss, mask matching loss, and security semantic loss, respectively. For the first Sample The probability distribution of the corresponding real image caption text description. For the first Sample The probability distribution of the corresponding predicted image caption text description. The DICE loss is used to generate the mask pixel set and the real mask pixel set for the segmentation mask decoding submodule. To generate the IOU loss between the mask pixel set and the real mask pixel set for the segmentation mask decoding submodule, These are the weighting coefficients for the DICE loss. For cosine similarity, The text description for the image captions with safety hazard markers generated by the text generation submodule. This refers to the relevant code clauses in the construction safety code database that relate to safety hazard markings in the image captions and text descriptions. This is the semantic vector obtained after encoding by the BERT model.

Images