Construction site safety supervision agent system based on multi-mode large language model

Abstract

The invention discloses a construction site safety supervision agent system based on a multi-modal large language model. The construction site safety supervision agent system comprises a multi-modal fusion module, a task analysis module and a hybrid execution engine. The multi-modal fusion module extracts multi-modal features such as images / videos and acoustics, and the multi-modal features are aligned to an LLM semantic space through a projection layer; the task analysis module analyzes the alignment features to generate a task graph (nodes are sub-tasks, and edges are dependency relationships); and the hybrid execution engine dynamically fuses the features according to the task graph and drives the execution of the subtasks through a gating attention mechanism. Cross-modal semantic alignment is realized through multi-modal and LLM deep fusion, the complex task understanding ability is improved through task graph structured analysis, gating attention dynamically deploys feature resources, the execution efficiency and flexibility of the system are enhanced, the method is suitable for space task automatic processing of scenes such as intelligent robots and smart homes, and the method has wide application prospects. And an efficient scheme is provided for constructing a multi-modal interaction and autonomous decision-making agent.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to a construction site safety monitoring intelligent agent system based on a multimodal large language model. Background Technology

[0002] Artificial intelligence technologies, represented by deep learning, have achieved remarkable results in many fields such as computer vision, natural language processing, and recommender systems. However, building a deep neural network is a complex process, typically involving multiple steps such as data processing, model development, and model training. Current research relies heavily on expert design for deep neural networks, which is time-consuming and labor-intensive. Moreover, expert-designed networks are limited by the experts' own knowledge and experience, often resulting in suboptimal network structures.

[0003] Currently, deep learning networks primarily process data through tasks during operation. For example, the invention patent with announcement number CN 112381215 B, entitled "An Adaptive Search Space Generation Method and Apparatus for Automated Machine Learning," discloses the process of forming a set of visual tasks, determining the type of visual task, and then processing the task accordingly. However, when faced with complex tasks (i.e., multimodal data processing), the aforementioned patent cannot accurately determine the type of task. Therefore, the method and apparatus of this patent are clearly unsuitable for handling complex tasks. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a construction site safety monitoring intelligent agent system based on a multimodal large language model that is applicable to complex tasks.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a construction site safety monitoring intelligent agent system based on a multimodal large language model, characterized in that it includes: The multimodal fusion module accepts multimodal data input from the user, extracts image / video features V through a visual encoder, extracts acoustic features A through an audio encoder, and then passes the data through a projection layer. , Alignment to the semantic space of LLM is defined by the following formula: Where T represents text input. , Trainable parameters The task parsing module based on LLM parses the features aligned to the semantic space of LLM by the multimodal fusion module, and then generates a task graph. In this task graph, nodes represent subtasks and edges represent dependencies. The specific formula is as follows: Let the task complexity be C, the number of subtasks be N, and the weight coefficients be... Indicates the priority of subtasks, satisfying... ,in, This is the function to execute for the subtask. The hybrid execution engine determines whether a subtask is to be executed locally or by calling a cloud API, and then drives the execution of the subtask.

[0006] As a further improvement of the present invention, the specific steps of the multimodal fusion module in extracting features and aligning and fusing them are as follows: Step 1: Extract video frame features from the input video data using lightweight ViT. And introduce the temporal dimension of temporal convolutional encoding: Obtain visual feature encoding, and output the input audio data using a Mel-spectrum-based Transformer encoder. Alignment with visual feature sampling in sync: Obtain audio feature encoding, and fine-tune the input text command using LoRA to obtain the text command encoding; Step 2: Input the visual feature encoding, audio feature encoding, and text instruction encoding obtained in Step 1 into the spatiotemporal alignment module for alignment, then into the fusion layer for fusion, and finally into the joint inference layer for joint inference, outputting cross-modal prediction results and simultaneously calculating the loss function.

[0007] As a further improvement of the present invention, the loss function in step two is as follows: The loss function is a joint optimization of mode alignment and task objective: In the formula, For cross-modal contrast loss, For time-series consistency loss, , and These are the weighting coefficients.

[0008] As a further improvement of the present invention, the specific steps of the LLM-based task parsing module in parsing features and generating a task graph are as follows: Step 3: Evaluate and output the task complexity using the LLM task parser; Step 4: Generate a hierarchical task graph; Step 5: Prioritize the subtasks in the generated hierarchical task graph, update the subtask weights based on the environmental feedback F, and update the hierarchical task graph synchronously based on the subtask dependencies. Step 6: Generate a task execution queue based on the latest task graph, and execute tasks according to the task execution queue by calling the tool library; Step 7: Integrate and provide feedback on the task execution results.

[0009] As a further improvement to the present invention, the specific method for task complexity evaluation in step three is as follows: The complexity of user input task T is jointly determined by semantic depth and modal requirements: In the formula, Here, M represents the weighting coefficients, and M is the set of modalities required for the task. Let m be the weight of mode m.

[0010] As a further improvement to the present invention, the specific method for generating the hierarchical task graph in step four is as follows: A probabilistic graphical model is used to construct subtask dependencies, with nodes representing subtasks. edge weight express arrive Transition probability: In the formula, This is the LLM scoring function for the relevance of subtasks.

[0011] As a further improvement to the present invention, the specific method for updating the subtask weights based on the environmental feedback F in step five is as follows: The learning rate controls the strength of the feedback's influence on priority. The old weights.

[0012] As a further improvement of the present invention, the specific steps for the hybrid execution engine to execute subtasks are as follows: Step 8, subtasks Make a routing decision: execute locally or call the cloud API. If you execute locally, proceed to step nine; if you call the cloud API, proceed to step ten. Step nine: If the execution is local, the tool will be invoked through the dynamic adapter, and resource consumption will be evaluated. Step 10: If it is a call to a cloud API, generate API request parameters, call the tool to execute, and perform a resource consumption assessment. Step 11: Combine the resource consumption assessment results from Step 9 and Step 10 to optimize resources, and then feed the execution results back to the LLM-based task parsing module.

[0013] As a further improvement of the present invention, the routing decision in step eight is made in the following manner: Based on the subtask complexity C(S_i) and real-time computing resources R, decide whether to execute locally or call the cloud API: In the formula, Set the local processing complexity threshold.

[0014] As a further improvement to the present invention, the design method of the dynamic adapter in step nine is as follows: Insert a lightweight adapter module into the pre-trained LLM, fine-tuning only 5% of the parameters: In the formula, It is a two-layer neural network, with the hidden layer dimension being 1 / 10 of the original model.

[0015] The beneficial effects of this invention are: Solving the problem of complex task decomposition and execution: Through the Large Language Model (LLM), complex tasks input by users (such as "identify environmental hazards and provide voice prompts") are dynamically decomposed into sub-tasks such as visual perception, audio analysis, and logical reasoning.

[0016] Multimodal information fusion and interaction: Combining modal data such as images, videos, and audio, spatial perception and real-time feedback are achieved using ordinary cameras or low-cost sensors.

[0017] Lightweight and efficient computation: The computational complexity of multimodal fusion is reduced through cross-modal feature alignment and lightweight connection layer design. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the multimodal fusion and inference process; Figure 2 This is a diagram illustrating the closed-loop process of task breakdown, planning, and execution. Figure 3 This is a schematic diagram of a lightweight execution architecture. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the embodiments shown in the accompanying drawings.

[0020] This embodiment of a construction site safety monitoring intelligent agent system based on a multimodal large language model includes: The multimodal fusion module accepts multimodal data input from the user, extracts image / video features V through a visual encoder (ViT or CNN), extracts acoustic features A through an audio encoder (Transformer), and then passes the data through a projection layer. , Alignment to the semantic space of LLM is defined by the following formula: Where T represents text input. , Trainable parameters The task parsing module based on LLM parses the features aligned to the semantic space of LLM by the multimodal fusion module, and then generates a task graph. In this task graph, nodes represent sub-tasks (such as object detection and speech synthesis), and edges represent dependencies. The specific formula is as follows: Let the task complexity be C, the number of subtasks be N, and the weight coefficients be... Indicates the priority of subtasks, satisfying... ,in, This is the function to execute for the subtask. The hybrid execution engine employs a gated attention mechanism to dynamically fuse features based on the task graph generated by the LLM-based task parsing module, and then drives the execution of subtasks. In the process of executing complex tasks, multimodal instructions and data extraction features are first aligned to the semantic space of the LLM. Then, the task parsing module parses the aligned features to generate a task graph. Finally, the hybrid execution engine executes the task graph. Compared with existing technologies, this approach can effectively handle complex multimodal tasks.

[0021] The specific process is as follows: First, the Cross-modal Spatiotemporal Alignment (CSTA) module maps visual, audio, and text features to a unified semantic space, supporting spatiotemporal inference. Step 1: Extract video frame features from the input video data using lightweight ViT. And introduce the temporal dimension of temporal convolutional encoding: Obtain visual feature encoding, and output the input audio data using a Mel-spectrum-based Transformer encoder. Alignment with visual feature sampling in sync: Obtain audio feature encoding, and fine-tune the input text command using LoRA to obtain the text command encoding; Step 2: Input the visual feature encoding, audio feature encoding and text instruction encoding obtained in Step 1 into the spatiotemporal alignment module for alignment, then into the fusion layer for fusion, and finally into the joint inference layer for joint inference, outputting cross-modal prediction results and simultaneously calculating the loss function; Secondly, the LLM-based task parser transforms user instructions into executable hierarchical task graphs and dynamically adjusts subtask priorities based on environmental feedback. Step 3: Evaluate and output the task complexity using the LLM task parser; Step 4: Generate a hierarchical task graph; Step 5: Prioritize the subtasks in the generated hierarchical task graph, update the subtask weights based on the environmental feedback F, and update the hierarchical task graph synchronously based on the subtask dependencies. Step 6: Generate a task execution queue based on the latest task graph, and the hybrid execution engine calls the tool library to execute subtasks according to the task execution queue; Finally, during the execution of subtasks, a dynamic adapter and a subtask router are combined to achieve on-demand allocation of computing resources: Step 7, subtasks Make a routing decision: execute locally or call the cloud API. If you execute locally, proceed to step nine; if you call the cloud API, proceed to step ten. Step 8: If the execution is local, the tool will be invoked through the dynamic adapter, and resource consumption will be evaluated. Step 9: If it is a call to a cloud API, generate API request parameters, call the tool to execute, and perform a resource consumption assessment. Step 10: Combine the resource consumption assessment results from Step 9 and Step 10 to optimize resources, and then feed back the execution results (i.e., environmental feedback F) to the LLM-based task parsing module.

[0022] In each of the above steps, this embodiment further provides specific methods for each of the key steps: In steps one and two above: The spatiotemporal feature encoding is as follows: - Visual Features: Extracting Video Frame Features Using Lightweight ViT And introduce the temporal dimension of temporal convolutional encoding: - Audio characteristics: Mel spectrum-based Transformer encoder output Alignment with visual feature sampling in sync: Cross-modal projection and fusion are as follows: Align different modal features using a gated projection layer: - Sigmoid function Element-wise multiplication, Gated weight matrix (trainable parameters) The spatiotemporal inference loss function is as follows: Joint optimization of modal alignment and task objective: Cross-modal contrastive loss (maximizing the similarity of positive sample pairs) Temporal consistency loss (penalizing abrupt changes in features between adjacent frames).

[0023] In steps three through six above: Task complexity assessment: The complexity of user input task T is jointly determined by semantic depth and modal requirements: - Weighting coefficients (obtained from historical task data) - M: The set of modalities required for the task (visual, audio, text, etc.) - The weights of modality m (e.g., when the visual modality has a higher weight than the audio modality). ) Task graph generation algorithm: A probabilistic graphical model is used to construct subtask dependencies, with nodes representing subtasks. edge weight express arrive Transition probability: - LLM scoring function for the relevance of subtasks (based on semantic similarity and logical coherence) Dynamic priority adjustment: Update the subtask weights based on environmental feedback F (such as sensor data, execution results): - The learning rate controls the strength of the impact of feedback on priorities (see the self-correction mechanism in the reference).

[0024] In steps seven through ten above: Subtask routing strategy: Based on the subtask complexity C(S_i) and real-time computing resources R, decide whether to execute locally or call the cloud API: - Local processing complexity threshold (can be dynamically adjusted) Dynamic adapter design: Insert a lightweight adapter module into the pre-trained LLM, fine-tuning only 5% of the parameters: - A two-layer neural network, with hidden layer dimensions 1 / 10 of the original model. Resource optimization goals: Minimize total latency and energy consumption: - Subtask Execution time - Subtask Energy consumption (refer to the robot energy consumption model).

[0025] This embodiment provides the following typical application process example: Scenario: The intelligent security system executes the "detect abnormal intrusion and issue an alert" command. Task breakdown: - Subtask 1: Object detection in camera video stream (human / vehicle classification) - Subtask 2: Microphone audio analysis (sound recognition of broken glass) - Subtask 3: Multimodal Evidence Fusion and Threat Level Assessment - Subtask 4: Generation of voice / light warning signals Dynamic resource allocation: - Subtasks 1-2 are executed locally (low latency required) Subtask 3: Call the cloud-based PaLM-E API (high computational requirements) Execution result: - Humanoid target detected + sound of breaking glass → Threat level = High → Trigger 110 decibel alarm. Summary table of formulas and parameters used above. In summary, the construction site safety monitoring intelligent agent system based on a multimodal large language model in this embodiment has the following characteristics: Dynamic task decomposition mechanism: The task parsing module based on LLM generates a task graph based on user input, where nodes represent subtasks (such as object detection and speech synthesis) and edges represent dependencies. Formula: Let the task complexity be C, the number of subtasks after decomposition be N, and the weight coefficients be... Indicates the priority of subtasks, satisfying... ,in This is the function to execute the subtask. Multimodal fusion module: A visual encoder (ViT or CNN) extracts image / video features V, and an audio encoder (Transformer) extracts acoustic features A, which are then projected through a projection layer. , Aligned to the semantic space of the LLM, the fusion formula is: Where T represents text input. , These are trainable parameters. Lightweight cross-modal connectivity layer: We employ a gated attention mechanism, which only fine-tunes a small number of parameters (such as linear layers or adapters) to preserve the knowledge of the pre-trained LLM.

[0026] Compared to existing technologies, this approach can effectively achieve the parsing, processing, and execution of complex multimodal tasks.

[0027] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A construction site safety monitoring intelligent agent system based on a multimodal large language model, characterized in that: include: The multimodal fusion module accepts multimodal data input from the user, extracts image / video features V through a visual encoder, extracts acoustic features A through an audio encoder, and then passes the data through a projection layer. , Alignment to the semantic space of LLM is defined by the following formula: Where T represents text input. , Trainable parameters The task parsing module based on LLM parses the features aligned to the semantic space of LLM by the multimodal fusion module, and then generates a task graph. In this task graph, nodes represent subtasks and edges represent dependencies. The specific formula is as follows: Let the task complexity be C, the number of subtasks be N, and the weight coefficients be... Indicates the priority of subtasks, satisfying... ,in, This is the function to execute for the subtask. The hybrid execution engine determines whether a subtask is to be executed locally or by calling a cloud API, and then drives the execution of the subtask.

2. The construction site safety monitoring intelligent agent system based on a multimodal large language model according to claim 1, characterized in that: The specific steps for feature extraction and alignment fusion by the multimodal fusion module are as follows: Step 1: Extract video frame features from the input video data using lightweight ViT. And introduce the temporal dimension of temporal convolutional encoding: Obtain visual feature encoding, and output the input audio data using a Mel-spectrum-based Transformer encoder. Alignment with visual feature sampling in sync: Obtain audio feature encoding, and fine-tune the input text command using LoRA to obtain the text command encoding; Step 2: The visual feature encoding, audio feature encoding, and text instruction encoding obtained in Step 1 are input into the spatiotemporal alignment module for alignment. The alignment method is to use a gated projection layer to align different modal features. Then, the input is fed into the fusion layer for fusion, and finally fed into the joint inference layer for joint inference. The cross-modal prediction result is output, and the loss function is calculated at the same time.

3. The construction site safety monitoring intelligent agent system based on a multimodal large language model according to claim 2, characterized in that: The loss function in step two is as follows: The loss function is a joint optimization of mode alignment and task objective: In the formula, For cross-modal contrast loss, For time-series consistency loss, , and These are the weighting coefficients.

4. The construction site safety monitoring intelligent agent system based on a multimodal large language model according to any one of claims 1 to 3, characterized in that: The specific steps of the LLM-based task parsing module in parsing features and generating a task graph are as follows: Step 3: Evaluate and output the task complexity using the LLM task parser; Step 4: Generate a hierarchical task graph; Step 5: Prioritize the subtasks in the generated hierarchical task graph, update the subtask weights based on the environmental feedback F, and update the hierarchical task graph synchronously based on the subtask dependencies. Step 6: Generate a task execution queue based on the latest task graph, and the hybrid execution engine calls the tool library to execute tasks according to the task execution queue.

5. The construction site safety monitoring intelligent agent system based on a multimodal large language model according to claim 4, characterized in that: The specific method for task complexity evaluation in step three is as follows: The complexity of user input task T is jointly determined by semantic depth and modal requirements: In the formula, Here, M represents the weighting coefficients, and M is the set of modalities required for the task. Let m be the weight of mode m.

6. The construction site safety monitoring intelligent agent system based on a multimodal large language model according to claim 5, characterized in that: The specific method for generating the hierarchical task graph in step four is as follows: A probabilistic graphical model is used to construct subtask dependencies, with nodes representing subtasks. edge weight express arrive Transition probability: In the formula, This is the LLM scoring function for the relevance of subtasks.

7. The construction site safety monitoring intelligent agent system based on a multimodal large language model according to claim 6, characterized in that: The specific method for updating the subtask weights based on environmental feedback F in step five is as follows: The learning rate controls the strength of the feedback's influence on priority. The old weights.

8. The construction site safety monitoring intelligent agent system based on a multimodal large language model according to any one of claims 1 to 3, characterized in that: The specific steps for the hybrid execution engine to execute subtasks are as follows: Step 7, subtasks Make a routing decision: execute locally or call the cloud API. If you execute locally, proceed to step nine; if you call the cloud API, proceed to step ten. Step 8: If the execution is local, the tool will be invoked through the dynamic adapter, and resource consumption will be evaluated. Step 9: If it is a call to a cloud API, generate API request parameters, call the tool to execute, and perform a resource consumption assessment. Step 10: Combine the resource consumption assessment results from Step 9 and Step 10 to optimize resources, and then feed the execution results back to the LLM-based task parsing module.

9. The construction site safety monitoring intelligent agent system based on a multimodal large language model according to claim 8, characterized in that: The routing decision in step eight is made in the following manner: Based on the subtask complexity C(S_i) and real-time computing resources R, decide whether to execute locally or call the cloud API: In the formula, Set the local processing complexity threshold.

10. The construction site safety monitoring intelligent agent system based on a multimodal large language model according to claim 9, characterized in that: The design method for the dynamic adapter in step nine is as follows: Insert a lightweight adapter module into the pre-trained LLM, fine-tuning only 5% of the parameters: In the formula, It is a two-layer neural network, with the hidden layer dimension being 1 / 10 of the original model.

Images