A power operation safety monitoring and question-answering method and system based on a multi-modal large model

Abstract

This invention discloses a method and system for power operation safety monitoring and question answering based on a multimodal large model. The method includes the following steps: a data acquisition step, acquiring visual data and safety regulation text data from the power operation site; a multimodal semantic fusion step, generating a unified fused semantic vector through feature extraction and cross-modal alignment fusion technology; a knowledge reasoning step, matching the fused semantic vector with a pre-constructed power safety knowledge graph and performing compliance reasoning based on a graph neural network; and an intelligent question answering generation step, inputting the fused semantic vector and the reasoning results into a large language model to generate natural language question answering information. This invention, through the synergistic innovation of multimodal semantic fusion, knowledge graph reasoning, and a large language model, solves the technical problems of insufficient semantic understanding, disconnect between the question answering system and the field, and unstructured knowledge representation in existing technologies, achieving fully automated and interpretable intelligent safety supervision from perception to cognition.

Description

Technical Field

[0001] This invention relates to a method for monitoring and answering questions about work safety, and more particularly to a method and system for monitoring and answering questions about power work safety based on a multimodal large model. Background Technology

[0002] Safety in power operations is the cornerstone of stable power system operation, involving various high-risk scenarios such as substation maintenance and line operation and maintenance. With the development of computer vision and natural language processing technologies, intelligent safety monitoring has become an important research direction. However, existing technical solutions still have significant shortcomings in achieving truly intelligent supervision, mainly in the following two aspects:

[0003] First, mainstream computer vision-based methods for identifying work behaviors primarily rely on algorithms such as object detection and pose recognition to analyze video images. While these methods can identify basic behaviors like "personnel climbing" and "using tools," their understanding remains at the pixel and object level, lacking a deep semantic understanding of work intent and the compliance of operational procedures. This single-visual-modality approach lacks the ability to correlate on-site behavior with textual safety regulations, resulting in biased and uninterpretable judgments.

[0004] Secondly, safety regulation question-and-answer systems based on knowledge retrieval rely on natural language processing technology to retrieve matching answers from regulatory documents. While these systems can provide regulatory clause lookup services, their information sources are limited to static text databases, failing to perceive real-time dynamic information at the work site. When on-site personnel ask, "Is my current operation compliant?", such systems cannot answer because they cannot acquire and understand the on-site context. Visual monitoring and knowledge question-and-answer systems operate independently, forming "information silos" that cannot achieve cross-modal semantic alignment and intelligent interaction in complex and dynamic scenarios.

[0005] In summary, existing power operation safety monitoring technologies suffer from three major shortcomings: a lack of semantic understanding, cross-modal information isolation, and uninterpretable knowledge judgments. Single-mode visual perception or text retrieval is insufficient to meet the demands of intelligent safety management for a complete "perception-cognition-decision-interpretation" chain. Therefore, there is an urgent need for a new intelligent monitoring and question-answering method that can integrate multimodal information, combine domain knowledge, and perform interpretable reasoning. Summary of the Invention

[0006] Purpose of the invention: The first purpose of this invention is to provide an intelligent power operation safety monitoring and question-and-answer method that can integrate multimodal information, combine domain knowledge, and perform interpretable reasoning; another purpose of this invention is to provide a system for implementing this method.

[0007] Technical solution: The power operation safety monitoring and question-and-answer method based on a multimodal large model described in this invention includes the following steps:

[0008] (1) Data acquisition: Acquire visual data of the power operation site and text data related to power operation safety;

[0009] (2) Multimodal semantic fusion: Process the visual data and text data, and extract the corresponding visual feature vectors and text feature vectors respectively; map the visual feature vectors and text feature vectors to a unified semantic space for cross-modal semantic alignment and fusion, and output the fused semantic vector;

[0010] (3) Knowledge reasoning: The fused semantic vector is matched with the pre-constructed power safety knowledge graph, and reasoning is performed based on the graph neural network to obtain reasoning results that include compliance judgments and their basis;

[0011] (4) Intelligent question and answer output: The fused semantic vector and the reasoning result are input into the large language model to generate and output interpretable natural language question and answer information, wherein the question and answer information is associated with on-site behavior, safety regulations and judgment conclusions.

[0012] Preferably, the visual data in step (1) includes video streams and image sequences of the work site, and the text data includes power safety regulations and work permit texts.

[0013] Preferably, in step (2), the visual feature vector and the text feature vector are semantically aligned through contrastive learning, and feature fusion is achieved through an attention mechanism.

[0014] Preferably, the semantic alignment of visual features and textual features is optimized using the InfoNCE loss function.

[0015] Preferably, in step (2), a visual encoder is used to extract visual feature vectors; and a text encoder is used to extract text feature vectors.

[0016] Preferably, the visual encoder is based on the Transformer architecture, and the text encoder is a pre-trained language model.

[0017] Preferably, the knowledge graph in step (3) includes power safety regulations, operational entities, and their logical relationships, represented in the form of triples.

[0018] Preferably, the graph neural network in step (3) is a graph attention network.

[0019] Preferably, the large language model in step (4) is the Qwen series model.

[0020] The power operation safety monitoring and question-and-answer system based on a multimodal large model described in this invention includes:

[0021] The data acquisition module is configured to acquire visual and text data from the power operation site.

[0022] A multimodal semantic fusion module is communicatively connected to the data acquisition module and is configured to process the visual data and text data to extract feature vectors, perform cross-modal semantic alignment and fusion, and output a fused semantic vector.

[0023] The knowledge reasoning module is communicatively connected to the multimodal semantic fusion module and is configured to match and reason with the fused semantic vector and the pre-built power safety knowledge graph to obtain the reasoning result;

[0024] The intelligent question answering module is communicatively connected to the knowledge reasoning module and is configured to input the fused semantic vector and reasoning results into the large language model to generate interpretable natural language question answering information.

[0025] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: (1) It realizes cross-modal semantic level understanding and effectively solves the problems of data silos and semantic gaps; (2) It improves the accuracy and interpretability of violation identification and risk judgment; (3) It constructs a closed-loop automated supervision system from perception to decision-making. Attached Figure Description

[0026] Figure 1 This is the overall technical architecture of the present invention;

[0027] Figure 2 This is the timing sequence of the data interaction phase of the present invention. Detailed Implementation

[0028] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0029] As shown in the attached figures, the system of the present invention adopts a modular and layered design as a whole, consisting of a data acquisition module, a multimodal preprocessing module, a semantic alignment and fusion module, a knowledge reasoning module, and an intelligent question answering module.

[0030] 1. Data Acquisition and Preprocessing

[0031] This step corresponds to the data acquisition module and the multimodal preprocessing module in the system.

[0032] First, multi-source heterogeneous data is collected from the power operation site. The data includes videos of the operation site, image sequences extracted from the videos, and documents containing work permit information and power safety regulations.

[0033] Next, data preprocessing is performed to standardize and vectorize data from different modalities.

[0034] (1) Visual data processing

[0035] The acquired image sequence is input into a visual converter for feature extraction. In this embodiment, the visual converter employs a Vision Transformer model based on the Transformer architecture. Through multi-layer Transformer encoder processing, a visual feature vector is finally output. :

[0036]

[0037] (2) Text data processing

[0038] Natural language processing methods are used to segment and sentence-by-sentence text data such as regulations and work orders. Named Entity Recognition (NER) technology is used to extract key entities. Subsequently, a text encoder is used to map the text information into vector representations. In this embodiment, the text encoder is a pre-trained language model BERT (Bidirectional Encoder Representations from Transformers), which can extract the contextual semantic representation of the text and output text feature vectors. :

[0039]

[0040] After the above processing steps, the final output is a standardized visual feature vector. With text feature vectors .

[0041] 2. Multimodal semantic fusion

[0042] This step corresponds to the semantic alignment and fusion module in the system. Its function is to use a multimodal large model to achieve cross-modal alignment and fusion of visual features and text semantics, forming a unified multimodal semantic representation.

[0043] (1) Feature mapping and normalization

[0044] First, the visual feature vector With text feature vectors A linear projection layer is used to transform and map the data to a shared semantic space. The linear projection layer maps visual and textual features to the same dimension through linear transformations. The mapping process is as follows:

[0045]

[0046]

[0047] in , For weight parameters, , These are the visual and textual features after transformation and mapping.

[0048] Subsequently, the LayerNorm module is used to standardize and regularize the mapped features to improve the model's robustness. The specific formula is as follows:

[0049]

[0050] in For feature vectors, The mean, Standard deviation For scaling parameters, This is the offset parameter.

[0051] (2) Cross-modal alignment

[0052] Semantic alignment is achieved using a contrastive learning strategy and optimized using the InfoNCE loss function, the calculation formula of which is as follows:

[0053]

[0054]

[0055]

[0056] in Represents cosine similarity. For temperature parameters, To calculate the image-to-text contrast loss, To calculate the text-image contrast loss, This mechanism calculates the average loss for image-to-text comparison and text-to-image comparison. It aims to maximize the similarity between matched image-to-text pairs and minimize the similarity between unmatched pairs, thus ensuring that the distance between true image-to-text pairs is minimized and the distance between incorrect pairs is maximized.

[0057] (3) Feature fusion

[0058] The aligned features are then deeply fused using a feature fusion module. This module employs a multi-head attention mechanism to calculate the relevance between visual and textual semantics, using textual features as the query and visual features as the key / value pair. The key information is fused through the attention mechanism, and the visual and textual semantic features are weighted and fused to obtain the fused feature representation.

[0059]

[0060]

[0061] in These are the aligned semantic features.

[0062] Aligned semantic features After further processing by a Transformer encoder consisting of a multi-head self-attention layer (MHA), a feedforward layer, and residual connections, the final output is a fused semantic vector. :

[0063]

[0064] 3. Knowledge Graph Reasoning

[0065] This step corresponds to the knowledge graph reasoning module of the system.

[0066] (1) Knowledge graph construction

[0067] A two-stage information extraction architecture based on a deep pre-trained model is adopted to construct a knowledge graph applicable to power safety operations, combining national standards and relevant safety regulations for the power industry. First, entity extraction is performed using a BERT Joint Conditional Random Field (CRF) model to identify entities such as operation type, violation clauses, equipment, and protective measures. Then, a BERT relation recognition model is used to extract relationships between entities, forming knowledge triples of (entity, relation, entity), and this is used to construct a structured knowledge graph for power safety operations.

[0068] (2) Knowledge reasoning

[0069] First, the fused multimodal semantic vectors With knowledge graph node vectors Similarity matching is performed, and the set of candidate nodes with the highest similarity is selected. The formula for calculating the matching similarity is:

[0070]

[0071] in Given the set of k candidate nodes with the highest similarity, To calculate cosine similarity.

[0072] Subsequently, starting from these nodes, inference is performed using a graph neural network. In this embodiment, a graph attention network (GAT) is used for inference. Through its information propagation mechanism, information is propagated within the subgraph containing the relevant nodes to obtain knowledge context representations related to the problem, enabling multi-hop logical reasoning and ultimately outputting the reasoning result. The propagation formula is:

[0073]

[0074] in Represented as nodes, This is the weight matrix. For activation function, For attention weights, For nodes The updated representation after dissemination.

[0075] 4. Intelligent question and answer output

[0076] This step corresponds to the intelligent question-and-answer module in the system.

[0077] This module is the system's output layer, which will fuse semantic vectors. With knowledge reasoning results The data is concatenated and input into a large language model. In this embodiment, the large language model is a Qwen series model. The model combines multimodal fusion features with structured knowledge graph context to parse the user's question intent in text form and generate a natural language answer containing judgment criteria for output, thereby realizing natural language-driven security monitoring, risk question answering, and reasoning decision-making.

[0078] In summary, through the above four core steps, this invention achieves fully automated intelligent supervision of the entire process of power operation site, from multimodal perception, semantic understanding, knowledge reasoning to interpretable question answering.

Claims

1. A method for power operation safety monitoring and question-and-answering based on a multimodal large model, characterized in that, Includes the following steps: (1) Data acquisition: Acquire visual data of the power operation site and text data related to power operation safety; (2) Multimodal semantic fusion: Process the visual data and text data, and extract the corresponding visual feature vectors and text feature vectors respectively; map the visual feature vectors and text feature vectors to a unified semantic space for cross-modal semantic alignment and fusion, and output the fused semantic vector; (3) Knowledge reasoning: The fused semantic vector is matched with the pre-constructed power safety knowledge graph, and reasoning is performed based on the graph neural network to obtain reasoning results that include compliance judgments and their basis; (4) Intelligent question and answer output: The fused semantic vector and the reasoning result are input into the large language model to generate and output interpretable natural language question and answer information, wherein the question and answer information is associated with on-site behavior, safety regulations and judgment conclusions.

2. The method according to claim 1, characterized in that, The visual data mentioned in step (1) includes video streams and image sequences of the work site, and the text data includes power safety regulations and work permit texts.

3. The method according to claim 1, characterized in that, In step (2), the visual feature vector and the text feature vector are semantically aligned through contrastive learning, and feature fusion is achieved through an attention mechanism.

4. The method according to claim 3, characterized in that, The semantic alignment of the visual features and text features is optimized using the InfoNCE loss function.

5. The method according to claim 1, characterized in that, In step (2), a visual encoder is used to extract the visual feature vector; a text encoder is used to extract the text feature vector.

6. The method according to claim 5, characterized in that, The visual encoder is based on the Transformer architecture, and the text encoder is a pre-trained language model.

7. The method according to claim 1, characterized in that, The knowledge graph described in step (3) contains electrical safety regulations, operational entities, and their logical relationships, represented in the form of triples.

8. The method according to claim 1, characterized in that, The graph neural network mentioned in step (3) is a graph attention network.

9. The method according to claim 1, characterized in that, The large language model mentioned in step (4) is the Qwen series model.

10. A power operation safety monitoring and question-and-answer system based on a multimodal large model, implementing the method as described in any one of claims 1 to 9, characterized in that, include: The data acquisition module is configured to acquire visual and text data from the power operation site. A multimodal semantic fusion module is communicatively connected to the data acquisition module and is configured to process the visual data and text data to extract feature vectors, perform cross-modal semantic alignment and fusion, and output a fused semantic vector. The knowledge reasoning module is communicatively connected to the multimodal semantic fusion module and is configured to match and reason with the fused semantic vector and the pre-built power safety knowledge graph to obtain the reasoning result; The intelligent question answering module is communicatively connected to the knowledge reasoning module and is configured to input the fused semantic vector and reasoning results into the large language model to generate interpretable natural language question answering information.

Images