Potential safety hazard real-time identification method and system based on multi-modal large model

Abstract

The invention belongs to the technical field of data processing, and particularly relates to a potential safety hazard real-time identification method and system based on a multi-modal large model, the system comprises the multi-modal large model, a field knowledge base and a multi-modal inference engine, the multi-modal large model is responsible for visual feature extraction and scene semantic understanding of an input field image, and the field knowledge base is responsible for field knowledge base analysis; processing a natural language query provided by a user; the domain knowledge base stores construction safety related laws and regulations, guidelines and historical cases, and factual basis is provided for the system through a structured storage and efficient retrieval mechanism; the multi-modal reasoning engine coordinates the whole process of visual understanding, task decomposition, knowledge retrieval and report generation, and is a core control module for realizing multi-modal reasoning and decision making. And refined understanding of entities and hidden dangers in a construction scene is realized. According to the system and the method thereof, the image content and the text specification can be dynamically fused, and missing detection or misjudgment caused by modal splitting in a traditional method is avoided.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular relates to a method and system for real-time identification of security risks based on a multimodal large model. Background Technology

[0002] Transformer: Transformer is a deep learning model proposed by Google researchers in 2017. It is mainly used to process sequential data. With the help of the Multi-Head Self-Attention (MHSA) mechanism to enhance the expressive power of the model, it has quickly become a core technology in the field of NLP.

[0003] ViT: short for VisionTransformer, is a model that applies the Transformer architecture to computer vision tasks. ViT was proposed by Google Research in 2020.

[0004] Self-Attention, also known as ScaledDot-ProductAttention, is an attention mechanism that allows a model to consider how each element in a sequence relates to the others when generating a representation of the sequence. Its core idea is that, given a sequence, the output at each position is a weighted sum of all positions in the sequence, with the weights (attention weights) calculated based on the correlation between the current position and each position in the sequence.

[0005] Token: A token usually refers to an element in the input sequence. In the NLP field, this element is a single word. In the vision field, an image is divided into a series of fixed-size image patches, and each image patch is regarded as a "token". BGE (BAAI General Embedding) model: A general text embedding model launched by Beijing Academy of Artificial Intelligence (BAAI) to convert text into high-dimensional vectors, which is widely used in tasks such as retrieval and semantic similarity calculation.

[0006] BM25: A classic probabilistic retrieval model used to estimate the relevance between a query and a document. It is one of the most commonly used and efficient sparse retrieval algorithms in the field of information retrieval.

[0007] Parameter Efficient Fine-Tuning (PEFT) technique: a type of technique designed to efficiently adapt large pre-trained models. Its core idea is to fine-tune only a small number of additional parameters instead of all model parameters, thereby significantly reducing computation and storage costs.

[0008] LoRA: A mainstream parameter efficient fine-tuning (PEFT) technique that approximates parameter updates by adding a low-rank decomposed adapter matrix to the model, thereby efficiently fine-tuning large models.

[0009] InternVL: A family of vision-language foundational models designed to align and match the capabilities of large vision models (such as InternViT) with those of large language models (such as LLM).

[0010] Contextual learning (ICL): A capability demonstrated by large language models (such as GPT-3) that allows the model to learn and complete new tasks simply by providing task examples (as context or cue words) without updating its parameters.

[0011] Thought Chain (CoT): A prompting technique used to improve the complex reasoning ability of large models by guiding them to solve problems by requiring them to generate a series of intermediate reasoning steps (i.e., "chains") rather than just the final answer.

[0012] Multilayer Perceptron (MLP): A basic feedforward neural network structure consisting of fully connected layers and stacked activation functions, it is one of the most fundamental components in deep learning.

[0013] InternViT: A powerful family of foundational vision models built on the VisionTransformer (ViT) architecture, pre-trained on large-scale datasets, and can serve as a powerful visual encoder for a variety of vision and multimodal tasks.

[0014] GPT models: A series of autoregressive language models developed by OpenAI, based on the Transformer decoder architecture, which have achieved breakthroughs in natural language processing tasks through generative pre-training.

[0015] Patches: In the visual Transformer model, this refers to dividing the input image into fixed-size image patches. These image patches are regarded as "tokens" of the sequence and are the basic units for the model to process images.

[0016] ColBERT: An efficient retrieval model that utilizes BERT for deep contextual encoding and achieves accurate and fast retrieval through a "delayed interaction" mechanism (i.e., encoding questions and documents independently first, and then performing lightweight interaction to calculate similarity).

[0017] The construction industry remains one of the world's most dangerous industries, with persistently high on-site injury and death rates. Although automated safety inspection technologies have emerged, the industry has long faced obstacles in real-time and effective monitoring of construction site safety. Current solutions often rely solely on visual data from the construction scene or process only text data, frequently isolating these two modalities. This "island-like" approach leads to incomplete or even inaccurate safety assessments, allowing potential hazards to escape detection and directly increasing worker risks.

[0018] The core challenge lies in how to integrate two types of heterogeneous information—visual observation and complex, ever-evolving text—into a collaborative and adaptive security detection system. Existing research has attempted this: extracting construction images into scene graphs and mapping them to textual security rules; using ontology to map visual and textual data; or leveraging dense image descriptions to compare visual scenes with security rules.

[0019] However, existing systems generally lack flexibility: they require large-scale retraining and manual updates to address new vulnerabilities or text adjustments. Furthermore, their understanding of visual and textual information is superficial, simplifying visual information into keywords or single sentences and covering only a few simple rules, making it difficult to handle complex and continuously updated text requirements.

[0020] LLMs have demonstrated exceptional capabilities in understanding and generating language, and recent advancements have extended these capabilities to multimodal scenarios, integrating visual and textual information. The effectiveness of multimodal methods in accident recognition and medical image diagnosis has been validated in fields such as transportation and healthcare. In the construction industry, the use of multimodal LLMs to generate construction inspection reports also shows considerable promise. Therefore, introducing multimodal LLMs is expected to dynamically integrate visual and textual data, improving the adaptability and comprehensiveness of safety inspections.

[0021] Nevertheless, directly transferring general-purpose LLMs and multimodal models to building safety inspection still faces significant obstacles: building safety data is extremely scarce in the massive training corpora of LLMs, resulting in insufficient model awareness of industry terminology, operating procedures, and risk characteristics. Furthermore, building safety inspection requires deep coupling of constantly evolving safety texts with sophisticated visual analysis of the dynamic construction environment, necessitating complex reasoning to identify minute hazards and compliance deficiencies that are difficult to detect visually alone.

[0022] In summary, existing building safety inspection technologies have the following main shortcomings: 1. Visual and textual modal separation: Traditional methods (such as image description generation or knowledge graphs) can only process images or security specifications separately and cannot deeply integrate the two, resulting in missed detections or false detections.

[0023] 2. Poor adaptability: New security risks require the collection of a large amount of labeled data and retraining of the model, which is costly and time-consuming.

[0024] 3. Low output credibility: Large language models are prone to "illusion" when lacking domain knowledge constraints, citing non-existent or incorrect specification provisions, which poses a significant security risk.

[0025] Therefore, a new method is urgently needed to deeply integrate advanced AI technology with domain knowledge to achieve highly reliable security checks. The core research question arising from this is: "How to design a multimodal method to dynamically fuse fine-grained visual data with complex, continuously evolving textual security specifications, thereby achieving adaptive security checks?" To address this problem, this invention proposes a novel multimodal security check method: bridging the visual-textual information gap by using Large Language Models (LLMs) and multimodal technology as its core. On the visual side, the InternVL architecture is employed to obtain pixel-level fine-grained understanding; on the knowledge retrieval side, the ColBERTlate-interaction model is introduced on top of existing hybrid retrieval to enhance fine-grained matching of long text clauses; on the reasoning side, a Tree-of-Thought framework is used for multi-step reflective reasoning in complex scenarios, significantly improving robustness. This method ultimately performs context-rich security analysis on construction site images, incorporating comprehensive security text and guidelines; and introduces a domain knowledge base, enabling the system to adaptively evolve with updates to security specifications, ensuring accurate and reliable output. Summary of the Invention

[0026] The purpose of this invention is to provide a real-time identification method and system for security risks based on a multimodal large model, so as to solve the above-mentioned technical problems existing in the prior art.

[0027] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: Firstly, a real-time security vulnerability identification system based on a multimodal large model is provided, including a multimodal large model, a domain knowledge base, and a multimodal inference engine, wherein the multimodal large model is connected to the multimodal inference engine, and the multimodal inference engine is connected to the domain knowledge base; The multimodal large model is responsible for extracting visual features and understanding scene semantics from the input on-site images, while also processing natural language queries provided by the user. The domain knowledge base is used to store construction safety guidelines and historical cases, providing a basis for the system through structured storage and efficient retrieval mechanisms; The multimodal reasoning engine is used to coordinate the entire process of visual understanding, task decomposition, knowledge retrieval and report generation, and is the control module for realizing multimodal reasoning and decision-making.

[0028] Preferably, the multimodal large model includes an LLM model, which is an extension of the InternVL architecture and includes a visual encoder, a text encoder and decoder, a projection adaptation layer and a security instruction fine-tuning module. The visual encoder uses the InternViT-6B model, which segments the image into patches and encodes them using a Transformer structure to output a visual feature sequence. The text encoder and decoder employ a pre-trained language model based on Transformer for encoding text queries and generating security reports. The projection adaptation layer maps visual features to the semantic space of the language model through a multilayer perceptron, thereby achieving visual-language modality alignment. The safety instruction fine-tuning module sets up a safety instruction fine-tuning mechanism to perform supervised fine-tuning using instruction data from construction safety scenarios.

[0029] Preferably, the implementation process of the visual encoder is as follows: The image is divided into a 14×14 patch sequence, and a serialized image token is obtained through linear projection. This token is then fed into a multi-layer Transformer encoder for encoding. The processing formula is as follows: ; in, , represents the input image, PatchEmbed represents the image patch and linear projection layer, E pos This represents the learnable absolute positional encoding, and L represents the total number of Transformer layers.

[0030] Preferably, the implementation process of the multimodal large model is as follows: Image acquisition: Images were collected from multiple construction sites, covering different construction stages, equipment types, and work scenarios; Image annotation: Describe the image, and in addition to the objects and scene content, annotate whether there are any safety hazards, violations of terms and conditions, and suggested measures to construct an image-instruction structure; Image-instruction data formatting: Convert image-instruction pairs into an input-output format acceptable to the LLM model; let X be the user input and Y be the desired model output, an image-instruction tuning data instance can be represented as: ; in v Represents site images, q This indicates a random query by the user, requesting security analysis; "\ n " represents a newline character in plain text;" <stop>" is a separator string provided to the model to indicate the separation between user input and model output; "user:" and "assistant:" are strings representing the roles in the dialogue; Model fine-tuning: LoRA technology is used to efficiently fine-tune the LLM, requiring only a few parameter updates to adapt to construction safety tasks.

[0031] Preferably, the multimodal reasoning engine includes a context warm-up module, a mind tree framework, and a context learning module; The context warm-up module is used to combine the image features extracted by the multimodal encoder with the text context to provide the model with domain background information; The mind tree framework systematically addresses the multi-factor trade-offs and dynamic decision-making problems in security analysis by generating multiple reasoning paths in parallel, dynamically evaluating states, and using intelligent backtracking search. The context learning module enables the model to quickly adapt to building safety regulations using a small number of sample examples.

[0032] Preferably, the implementation process of the multimodal inference engine is as follows: Visual perception of user input: Detailed text descriptions of user-uploaded images are generated using the LMM model, capturing key elements in the scene, including building structure, human activities, and equipment status; Based on user questions and image descriptions, a set of focused subqueries are generated using an independent plain text language model. Each subquery corresponds to a specific security check dimension, including PPE compliance and mechanical operation standardization, and explicitly points to the specific knowledge to be retrieved in the knowledge base. Targeted knowledge retrieval and local reasoning based on subqueries: Knowledge retrieval: Each subquery retrieves standards and cases from the constructed domain knowledge base; Sub-conclusion generation: The LMM model combines the subquery with the retrieved specialized knowledge to generate an evaluation conclusion for the subtask; Integrate information from multiple sources to generate a comprehensive security report.

[0033] Preferably, the domain knowledge base is configured with a hybrid retrieval strategy, the specific process of which is as follows: Preliminary Search: BM25 and ColBERT are used simultaneously to retrieve candidate documents from the knowledge base. BM25 quickly filters out documents containing the query keywords, while ColBERT retrieves documents that match the query semantics in parallel. Score fusion: The documents and scores returned by BM25 and ColBERT are standardized and then fused using a weighted sum. The specific formula is as follows: ; in, α This is the weighting coefficient, usually taken as 0.5. μ and These are the mean and standard deviation of the corresponding scores on the candidate set; Reordering and Returning: Sort candidate documents according to fusion scores and return the Top-K most relevant documents for subsequent response generation.

[0034] Secondly, a real-time security hazard identification method based on a multimodal large model is provided, which is implemented based on the aforementioned real-time security hazard identification system based on a multimodal large model, and includes the following steps: S1: Receives on-site images and natural language queries; S2: Generate image scene descriptions using LMM; S3: Based on the scenario description and user query, multiple security check sub-tasks are decomposed; S4: For each subtask, retrieve relevant security provisions from the domain knowledge base and generate sub-conclusions; S5: Integrate visual information with all sub-conclusions to generate the final security assessment report.

[0035] The beneficial effects of this invention include: The present invention provides a method and system for real-time identification of security risks based on a multimodal large model. 1. It achieves deep integration of visual and textual information, improving the accuracy and completeness of security checks: Existing methods often employ isolated image or text processing approaches, making it difficult to achieve fine-grained alignment across modalities. This invention constructs a vision-language joint encoding architecture based on a multimodal large language model (LMM), uses InternViT for pixel-level image encoding, and combines LoRA fine-tuning with a visual instruction dataset to achieve a refined understanding of entities and potential hazards in construction scenarios. It can dynamically fuse image content and text specifications, avoiding missed detections or misjudgments caused by modal fragmentation in traditional methods.

[0036] 2. It possesses good adaptability and scalability, supporting dynamic text updates and few-shot learning: Traditional systems require re-annotation and retraining when faced with new vulnerabilities or text changes, resulting in high costs. This invention constructs a scalable domain knowledge base and combines semantic segmentation with hybrid retrieval techniques (BM25+ColBERT) to achieve fine-grained matching and rapid retrieval of long text specification clauses. The system requires only a small number of samples (LoRA fine-tuning updates only 0.1% of parameters) to adapt to new specifications, significantly reducing iteration costs and deployment barriers.

[0037] 3. Enhance output credibility and interpretability through multi-step reasoning mechanisms: Existing large language models are prone to "illusion" in specialized fields, with outputs lacking constraints. This invention introduces a multi-step inference engine, based on the Tree-of-Thought framework, to decompose complex scenarios into tasks, perform subquery inference, and generate comprehensive conclusions. Combined with the Prompt library (ICL and CoT), it achieves a traceable inference path. This mechanism significantly improves the logical consistency and reliability of compliance judgments, making it suitable for high-risk building safety scenarios. Attached Figure Description

[0038] Figure 1 This is a structural diagram of the real-time safety hazard identification system based on a multimodal large model according to the present invention.

[0039] Figure 2 This is a schematic diagram of the LMM model of the present invention.

[0040] Figure 3 This is a schematic diagram of the hybrid retrieval process of the present invention.

[0041] Figure 4 This is a schematic diagram of the task decomposition and comprehensive reasoning process of the present invention. Detailed Implementation

[0042] The following is in conjunction with the appendix Figures 1-4 The present invention will be further described in detail below: See appendix Figure 1 As shown, a real-time security vulnerability identification system based on a multimodal large model includes a multimodal large model, a domain knowledge base, and a multimodal inference engine. The multimodal large model is connected to the multimodal inference engine, and the multimodal inference engine is connected to the domain knowledge base.

[0043] The multimodal large model is responsible for extracting visual features and understanding scene semantics from the input on-site images, while also processing natural language queries provided by the user. The domain knowledge base stores construction safety guidelines and historical cases, providing factual evidence to the system through structured storage and efficient retrieval mechanisms. The multimodal reasoning engine coordinates the entire process of visual understanding, task decomposition, knowledge retrieval, and report generation, serving as the control module for multimodal reasoning and decision-making.

[0044] In this embodiment, the multimodal large model includes an LLM model, which is an extension of the InternVL architecture and includes a visual encoder, a text encoder and decoder, a projection adaptation layer, and a safety instruction fine-tuning module. The visual encoder uses the InternViT-6B model, supporting image input at resolutions up to 4K. It segments the image into patches and encodes them using a Transformer structure, outputting a sequence of visual features. The text encoder and decoder employ a pre-trained language model based on Transformer for encoding text queries and generating safety reports. The projection adaptation layer maps visual features to the semantic space of the language model using a multilayer perceptron (MLP), achieving visual-language modality alignment. The safety instruction fine-tuning module sets up a safety instruction fine-tuning mechanism, using supervised fine-tuning based on instruction data from construction safety scenarios to improve the model's understanding and generation capabilities of safety semantics.

[0045] To enable the language model to process and understand visual information from construction sites, this invention integrates the InternVL large-scale visual encoder InternViT-6B into the language model. Unlike the traditional VisionTransformer (ViT), InternViT-6B is the core visual encoder in the InternVL framework, with 5.9B parameters, adopting the standard ViT architecture, and processing images by dividing them into 14×14 patch sequences.

[0046] The implementation process of the visual encoder is as follows: The image is divided into a 14×14 patch sequence, and a serialized image token is obtained through linear projection. This token is then fed into a multi-layer Transformer encoder for encoding. The processing formula is as follows: ; in, , represents the input image, PatchEmbed represents the image patch and linear projection layer, E pos This represents the learnable absolute positional encoding, and L represents the total number of Transformer layers.

[0047] InternViT-6B was trained in the first phase through comparative learning with LLaMA-7B, using approximately 5 billion cleaned image-text pairs from multiple sources including LAION-en, LAION-multi, COYO, and Wukong. Its training objective was to align the visual and linguistic representation spaces, providing a foundation for subsequent multimodal tasks.

[0048] In this invention, the InternVL visual encoder is used as the core component for multimodal understanding. Thanks to its pre-training on large-scale multilingual text and image data, the InternViT-6B has powerful visual understanding capabilities, can handle fine-grained visual details, and supports multilingual text descriptions, making it suitable for complex scenarios such as construction sites.

[0049] Since InternViT-6B has achieved alignment with the language model's embedding space through joint optimization with QLLaMA during the training phase, its output visual features are naturally compatible with the language model. To enhance its adaptability in the field of building safety, a lightweight MLP projection layer is introduced after the visual encoder to map general visual features to a semantic space related to construction safety.

[0050] During fine-tuning, InternViT-6B remained frozen, and only the MLP projection layer was trained. The projection layer was fine-tuned using safety instruction data from construction sites to effectively transform visual information in images into semantic representations highly relevant to the safety instruction generation task, thereby improving the model's performance in this domain.

[0051] Example 2 Based on Example 1, the implementation process of the multimodal large model is as follows: Image acquisition: Images were collected from multiple construction sites, covering different construction stages, equipment types, and work scenarios; Image annotation: Describe the image, and in addition to the objects and scene content, annotate whether there are any safety hazards, violations of terms and conditions, and suggested measures to construct an image-instruction structure; Image-instruction data formatting: Convert image-instruction pairs into an input-output format acceptable to the LLM model; let X be the user input and Y be the desired model output, an image-instruction tuning data instance can be represented as: ; in v Represents site images, q This indicates a random query by the user, requiring security analysis; n " represents a newline character in plain text;" <stop>" is a separator string provided to the model to indicate the separation between user input and model output; "user:" and "assistant:" are strings representing the roles in the dialogue.

[0052] Model fine-tuning: Using an image-instruction dataset, the LLM is efficiently fine-tuned using LoRA (Low-Rank Adaptation) technology, requiring only a small number of parameter updates to adapt to construction safety tasks. LoRA aims to update only a small subset of weights in the model that have the greatest impact on the current task, thereby achieving targeted adjustments that enable the model to understand and generate text that conforms to the professional context and content of construction safety.

[0053] The LoRA mechanism is integrated into the Transformer architecture. Each Transformer layer typically includes a multi-head attention mechanism and a feedforward network, with each module followed by an addition and normalization step. The LoRA adapter is strategically embedded in the feedforward network, allowing it to adjust the module's behavior during fine-tuning without significantly altering the original pre-trained weights. This process begins with the input vector, i.e., the output of the previous Transformer layer, which has a dimension of d. The d×d weights in the pre-trained model remain frozen during fine-tuning to ensure that the model's learned fundamental knowledge is not compromised.

[0054] Meanwhile, LoRA introduces two smaller trainable matrices WA∈R^(d×r) and WB∈R^(r×d), whose rank r is chosen according to the required level of adaptation for the task, to achieve a balance between flexibility and parameter efficiency. The change in weights ΔW in the feedforward network can be approximated as the product of WA and WB, which allows the model to efficiently adapt to new tasks without requiring full retraining.

[0055] Finally, the output vector is obtained by processing the input vector through the modified feedforward network. Specifically, the LoRAadapter first applies the weight updates to the input, and then adds the adjusted result to the output generated by the original frozen weights.

[0056] By carefully applying LoRA during the fine-tuning phase, the GPT model adapts to the specific needs of the building safety field while retaining the broad language understanding capabilities acquired during the pre-training phase. This balance enables the model to be both highly specialized and possess a comprehensive knowledge base when dealing with the complex and ever-changing tasks in building safety management.

[0057] The multimodal inference engine includes a context warm-up module, a thought tree (ToT) framework, and a context learning module. The context warm-up module combines image features extracted by a multimodal encoder (such as InternViT) with textual context, providing the model with rich domain background information. The thought tree (ToT) framework breaks through the limitations of traditional single-path inference, systematically handling multi-factor trade-offs and dynamic decision-making problems in safety analysis through parallel generation of multiple inference paths, dynamic state evaluation, and intelligent backtracking search. The context learning (ICL) module enables the model to quickly adapt to building safety regulations using few sample examples, ensuring that the output conforms to industry standards and possesses context relevance.

[0058] The implementation process of the multimodal inference engine is as follows: Visual perception of user input: Generate detailed text descriptions of user-uploaded images using the LMM model, and capture key elements in the scene, including building structure, human activities and equipment status. Based on user questions and image descriptions, to avoid visual model bias, an independent plain text language model is used to generate a set of focused subqueries. These subqueries correspond to specific safety inspection dimensions, including PPE compliance and mechanical operation standardization, and explicitly point to the specific knowledge to be retrieved in the knowledge base, thereby achieving transparent and refined decomposition of complex tasks.

[0059] Targeted knowledge retrieval and local reasoning based on subqueries: Knowledge retrieval: Each subquery retrieves standards and cases from the constructed domain knowledge base.

[0060] Sub-conclusion generation: The LMM model combines subqueries with retrieved specialized knowledge to generate evaluation conclusions for the specific subtask, such as "failure to wear a safety helmet is a violation according to Article Y of Standard XX". This process ensures that the judgment for each safety dimension is based on authoritative knowledge and that the reasoning process is traceable.

[0061] Integrate information from multiple sources to generate a comprehensive security report: Multimodal information fusion: The LMM model simultaneously receives visual content and all sub-conclusions, establishing a connection between scene context and professional judgment; Comprehensive reasoning: The model weighs the priorities and relevance among sub-conclusions, identifies risk patterns such as multi-risk concurrent scenarios, and supplements potential problems discovered by visual analysis; Report generation: Outputs a structured security assessment report, including a risk summary, compliance status, supporting text, and targeted improvement recommendations, ensuring the results are comprehensive and actionable.

[0062] The domain knowledge base employs a hybrid retrieval strategy, the specific process of which is as follows: Preliminary retrieval: Both the sparse retrieval component BM25 and the dense retrieval component ColBERT delayed interaction model are used simultaneously to retrieve candidate documents from the knowledge base. BM25 quickly filters out documents containing the query keywords, while ColBERT retrieves documents that semantically match the query in parallel. Score fusion: The documents and scores returned by BM25 and ColBERT are standardized and then fused using a weighted sum. The specific formula is as follows: ; in, α This is the weighting coefficient, usually taken as 0.5. μ and These are the mean and standard deviation of the corresponding scores on the candidate set; Reordering and Returning: Sort candidate documents according to fusion scores and return the Top-K most relevant documents for subsequent response generation.

[0063] The hybrid retrieval strategy combines preliminary sparse vector retrieval with dense multi-vector retrieval techniques based on the ColBERT (Contextualized Late Interaction over BERT) delayed interaction model. This hybrid retrieval method, rather than relying solely on simple dense vector cosine similarity, aims to simultaneously capture the accuracy of word matching and deep semantic similarity during the retrieval process, and utilizes the delayed interaction mechanism to achieve finer-grained relevance calculation.

[0064] Sparse vector retrieval excels at identifying documents containing exact matches or near-variants of query terms, ensuring lexical relevance. BM25, as a sparse retrieval component, calculates its similarity score as follows: ; in: q i It is the first i One query term, yes q i In the document d Frequency of occurrence in | d | is a document d Word length in avgdl It is the average length of the documents in the collection. k 1 and b These are free parameters, and are usually selected. k 1∈[1.2,2.0] and b=0.75, IDF ( q i )yes q i Inverse document frequency, N It is the total number of documents in the collection. n ( q i ) is included q i The number of documents.

[0065] ColBERT (Contextualized LateInteraction over BERT), as a dense retrieval component, is a BERT-based late-interaction model that achieves efficient and accurate retrieval by independently encoding queries and documents and performing lightweight but fine-grained interactions in the final stage.

[0066] How ColBERT works: ColBERT's core is a delayed interaction mechanism, which differs from traditional "early interaction" models (such as BERT's [CLS] concatenation of input): Early interaction models required concatenating the query and document before inputting them into BERT, resulting in computational complexity that increased linearly with document length and made it impossible to precompute document representations offline.

[0067] ColBERT Deferred Interaction Model: Queries and documents are independently encoded into two sets of contextualized token embedding vectors (preserving all token-level outputs of BERT), and the interaction is deferred to the final stage, with fine-grained similarity calculated through the MaxSim operator.

[0068] The specific process is as follows: Encoding phase: Query encoder f Q : will query ( q Encoded as a fixed-size set of embeddings The query input format is as follows: If the length is insufficient, pad with [MASK] to reach the fixed length. N q =32, reduced to 128 dimensions through a linear layer and then normalized by L2.

[0069] Document encoder f D : Document d Encoded as an embedded collection The document input format is as follows: (No padding), the embeddings corresponding to punctuation marks are filtered to reduce redundancy, and the dimensions are also reduced to 128 and normalized.

[0070] Interaction and scoring phase: ColBERT calculates relevance scores The formula is: ; In the formula, m and n are the number of tokens in the query and the document, respectively, and are not fixed values. The cosine similarity is achieved through the dot product (because the vectors have been normalized). The MaxSim operation retrieves the most similar token in the document for each token in the query, reflecting soft matching, such as "car" matching "automobile".

[0071] A real-time safety hazard identification method based on a multimodal large model, implemented based on the aforementioned real-time safety hazard identification system based on a multimodal large model, includes the following steps: S1: Receives on-site images and natural language queries; S2: Generate image scene descriptions using LMM; S3: Based on the scenario description and user query, multiple security check sub-tasks are decomposed; S4: For each subtask, retrieve relevant security provisions from the domain knowledge base and generate sub-conclusions; S5: Integrate visual information with all sub-conclusions to generate the final security assessment report.

[0072] Example 3 We collected 5,000 images of a construction site in a certain province and manually labeled each image with potential safety hazards, forming a dataset containing 4,500 training images and 500 independent test images.

[0073] The real-time security hazard identification system based on a multimodal large model of this invention was compared with existing detection methods based on image description-semantic similarity, with the test environment remaining consistent. The main evaluation metrics used were the art-recognized Completeness, Correctness, Answer Relevancy, Faithfulness, and a custom Hazard-identification metric, with scores ranging from 0 to 1, where higher values ​​indicate better performance. The test results are shown in Table 1 below: Table 1 Comparison of Results between the Invention System and Existing Technology Systems As shown in Table 1, in the scenario of "lack of fall protection from heights," the system of this invention improves precision by approximately 17.6% and recall by approximately 16.8%; the overall inference fidelity is ≥0.95, significantly outperforming existing technologies. Furthermore, when safety regulations are updated, this invention can take effect within minutes simply by adding or replacing text clauses in the domain knowledge base, without requiring re-collection of training samples or retraining of the model, verifying the system's zero-retraining adaptive capability. The above experimental results demonstrate that this invention can achieve high accuracy, high reliability, and easy maintenance for automatic identification of safety hazards in complex and ever-changing construction sites.

[0074] Existing methods often employ isolated image or text processing approaches, making it difficult to achieve fine-grained alignment across modalities. This invention constructs a vision-language joint encoding architecture based on a multimodal large language model (LMM), uses InternViT for pixel-level image encoding, and combines LoRA fine-tuning with a visual instruction dataset to achieve a refined understanding of entities and potential hazards in construction scenarios. This method can dynamically fuse image content and text specifications, avoiding missed detections or misjudgments caused by modal fragmentation in traditional methods.

[0075] Traditional systems require re-annotation and retraining when faced with new vulnerabilities or text changes, resulting in high costs. This invention constructs a scalable domain knowledge base and combines semantic segmentation with hybrid retrieval techniques (BM25+ColBERT) to achieve fine-grained matching and rapid retrieval of long text specification clauses. The system requires only a small number of samples (LoRA fine-tuning updates only 0.1% of parameters) to adapt to new specifications, significantly reducing iteration costs and deployment barriers.

[0076] Existing large language models are prone to "illusion" in specialized fields, with outputs lacking constraints. This invention introduces a multi-step inference engine, based on the Tree-of-Thought framework, to decompose complex scenarios into tasks, perform subquery inference, and generate comprehensive conclusions. Combined with the Prompt library (ICL and CoT), it achieves a traceable inference path. This mechanism significantly improves the logical consistency and reliability of compliance judgments, making it suitable for high-risk building safety scenarios.

[0077] In summary, the real-time security hazard identification method and system based on a multimodal large model provided by this invention constructs a multimodal security inspection architecture: a loosely coupled system architecture based on "large language model-visual encoder (LMM) + dynamic knowledge base + step-by-step inference engine" to achieve real-time fusion analysis of image and text security specifications. LoRA-MLP cross-modal alignment mechanism: under the premise of freezing the InternViT visual encoder, visual features are mapped to the latent space of the language model only through low-rank adaptation (LoRA) + lightweight MLP projection, achieving domain adaptation of construction scenarios while maintaining general knowledge without loss. Knowledge enhancement through semantic-keyword hybrid retrieval: employing BM25 sparse retrieval and ColBERTlate-interaction dense retrieval for dual-path recall, semantically segmenting and BGE vectorizing the security specifications, enabling newly added or revised texts to be deployed within minutes without retraining. The three-level reasoning process of task decomposition, sub-conclusions, and comprehensive conclusions is as follows: First, the language model breaks down the user query and image description into several sub-tasks. Then, the knowledge base is searched to generate sub-conclusions. Finally, the sub-conclusions are integrated with the visual content to output an overall security assessment, which improves interpretability and accuracy.< / stop> < / stop>

Claims

1. A real-time safety hazard identification system based on a multimodal large model, characterized in that, It includes a multimodal large model, a domain knowledge base, and a multimodal inference engine, wherein the multimodal large model is connected to the multimodal inference engine, and the multimodal inference engine is connected to the domain knowledge base; The multimodal large model is responsible for extracting visual features and understanding scene semantics from the input on-site images, while also processing natural language queries provided by the user. The domain knowledge base is used to store construction safety guidelines and historical cases, providing a basis for the system through structured storage and retrieval mechanisms; The multimodal reasoning engine is used to coordinate the entire process of visual understanding, task decomposition, knowledge retrieval and report generation, and is the control module for realizing multimodal reasoning and decision-making.

2. The real-time safety hazard identification system based on a multimodal large model according to claim 1, characterized in that, The multimodal large model includes an LLM model, which is an extension of the InternVL architecture and includes a visual encoder, a text encoder and decoder, a projection adaptation layer and a security instruction fine-tuning module. The visual encoder uses the InternViT-6B model, which segments the image into patches and encodes them using a Transformer structure to output a visual feature sequence. The text encoder and decoder employ a pre-trained language model based on Transformer for encoding text queries and generating security reports. The projection adaptation layer maps visual features to the semantic space of the language model through a multilayer perceptron, thereby achieving visual-language modality alignment. The safety instruction fine-tuning module sets up a safety instruction fine-tuning mechanism to perform supervised fine-tuning using instruction data from construction safety scenarios.

3. The real-time safety hazard identification system based on a multimodal large model according to claim 2, characterized in that, The implementation process of the visual encoder is as follows: The image is divided into a 14×14 patch sequence, and a serialized image token is obtained through linear projection. This token is then fed into a multi-layer Transformer encoder for encoding. The processing formula is as follows: ； in, , represents the input image, PatchEmbed represents the image patch and linear projection layer, E pos This represents the learnable absolute positional encoding, and L represents the total number of Transformer layers.

4. The real-time safety hazard identification system based on a multimodal large model according to claim 2, characterized in that, The implementation process of the multimodal large model is as follows: Image acquisition: Images were collected from multiple construction sites, covering different construction stages, equipment types, and work scenarios; Image annotation: Describe the image, and in addition to the objects and scene content, annotate whether there are any safety hazards and suggested measures to construct an image-instruction structure; Image-instruction data formatting: Convert image-instruction pairs into an input-output format acceptable to the LLM model; let X be the user input and Y be the desired model output, an image-instruction tuning data instance can be represented as: ； in v Represents a site image. q This indicates a random query by the user, requiring security analysis. n This represents a newline character in plain text; <stop> This is a separator string provided to the model to indicate the separation between user input and model output; user: and assistant: are strings representing the roles in the dialogue;< / stop> Fine-tuning the model: LoRA technology is used to fine-tune the LLM and update parameters to adapt to construction safety tasks.

5. A real-time safety hazard identification system based on a multimodal large model according to claim 1, characterized in that, The multimodal reasoning engine includes a context warm-up module, a mind tree framework, and a context learning module; The context warm-up module is used to combine the image features extracted by the multimodal encoder with the text context to provide the model with domain background information; The mind tree framework systematically addresses the multi-factor trade-offs and dynamic decision-making problems in security analysis by generating multiple reasoning paths in parallel, dynamically evaluating states, and using intelligent backtracking search. The context learning module enables the model to quickly adapt to building safety regulations through sample examples.

6. The real-time safety hazard identification system based on a multimodal large model according to claim 4, characterized in that, The implementation process of the multimodal inference engine is as follows: Visual perception of user input: Detailed text descriptions of user-uploaded images are generated using the LMM model, capturing elements in the scene, including building structure, human activities, and equipment status; Based on user questions and image descriptions, a set of focused subqueries are generated using an independent plain text language model. Each subquery corresponds to a specific safety inspection dimension, including the standardization of mechanical operations, and explicitly points to the specific knowledge to be retrieved in the knowledge base. Targeted knowledge retrieval and local reasoning based on subqueries: Knowledge retrieval: Each subquery retrieves relevant standards and cases from the constructed domain knowledge base; Sub-conclusion generation: The LMM model combines subqueries with retrieved specialized knowledge to generate evaluation conclusions for sub-tasks; Integrate information from multiple sources to generate a comprehensive security report.

7. A real-time safety hazard identification system based on a multimodal large model according to claim 1, characterized in that, The domain knowledge base employs a hybrid retrieval strategy, the specific process of which is as follows: Preliminary search: BM25 and ColBERT are used simultaneously to retrieve candidate documents from the knowledge base. BM25 quickly filters out documents containing the query keywords, while ColBERT retrieves documents that match the query semantics in parallel. Score fusion: The documents and scores returned by BM25 and ColBERT are standardized and then fused using a weighted sum. The specific formula is as follows: ； in, α These are weighting coefficients. μ and These are the mean and standard deviation of the corresponding scores on the candidate set; Reordering and Returning: Sort candidate documents according to fusion scores and return the Top-K most relevant documents for subsequent response generation.

8. A real-time safety hazard identification method based on a multimodal large model, implemented based on the real-time safety hazard identification system based on a multimodal large model as described in any one of claims 1-7, characterized in that, Includes the following steps: S1: Receives on-site images and natural language queries; S2: Generate image scene descriptions using LMM; S3: Based on the scenario description and user query, multiple security check sub-tasks are decomposed; S4: For each subtask, retrieve relevant security provisions from the domain knowledge base and generate sub-conclusions; S5: Integrate visual information with all sub-conclusions to generate the final security assessment report.

Images