Power grid multi-modal data compliance monitoring method based on knowledge graph

Abstract

The invention relates to a power grid multi-modal data compliance monitoring method based on a knowledge graph, and the method comprises the following steps: S1, obtaining multi-source heterogeneous data of a power grid, carrying out the preprocessing, and generating structured and semantic data representation; s2, constructing an initial knowledge graph according to the produced structured and semantic data representation; s3, according to the initial knowledge graph, combining a time perception mechanism TGN to identify compliance standards and laws of a power grid operation environment changing with time, combining rule-based logical reasoning with a deep neural model through neural symbol fusion, realizing compliance judgment under a fuzzy condition, and constructing a knowledge agent; s4, based on the knowledge agent, performing multi-modal compliance analysis in combination with deep learning and symbolic reasoning to obtain an analysis result; and S5, based on an analysis result, in combination with a natural language generation technology, automatically generating an interpretation report for the violation event. According to the invention, dynamic intelligent compliance analysis is realized, and the intelligent and automatic level of power grid management is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of data monitoring, and in particular to a knowledge graph-based method for monitoring compliance of multimodal data in power grids. Background Technology

[0002] With the accelerating digitalization and intelligentization of modern power systems, power grid companies are accumulating and transmitting massive amounts of multi-source, heterogeneous data in operation scheduling, equipment maintenance, market transactions, and information exchange. This data encompasses real-time SCADA monitoring information, equipment operating parameters, maintenance records, dispatch instructions, electricity market transaction data, and text documents and image / video materials related to management systems and safety regulations. With the advancement of the energy internet and data interconnection and sharing, a large amount of power grid business data needs to be circulated and shared externally between enterprises, regions, and even across industries. Power grid data is diverse in type, sensitive in content, complex in structure, and frequently updated. Furthermore, the security levels, privacy attributes, and compliance requirements of different data types vary significantly. Therefore, a systematic and intelligent compliance monitoring mechanism is urgently needed to ensure the safe, legal, and controllable use of data during external data circulation.

[0003] However, existing data compliance monitoring methods primarily rely on static rule configuration and manual review, which have limited semantic understanding of data content. This makes it difficult to accurately identify the implicit business semantics, sensitive elements, and contextual relationships within power grid data, thus hindering the effective determination of whether data flow activities cross security red lines or violate management procedures. Furthermore, traditional methods struggle to handle real-time compliance assessments and policy adjustments in dynamic data flow scenarios, lacking intelligent support for complex scenarios such as data classification and management, de-identification verification, and cross-modal security audits. This not only leads to low monitoring efficiency, delayed response, and high false alarm rates, but also makes it difficult to promptly detect and trace illegal data flow activities. Summary of the Invention

[0004] To address the aforementioned issues, the present invention aims to provide a knowledge graph-based method for monitoring compliance of multimodal data in power grids, enabling dynamic and intelligent compliance analysis and significantly improving the intelligence and automation level of power grid management.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A knowledge graph-based method for compliance monitoring of multimodal data in power grids includes the following steps: S1: Acquire multi-source heterogeneous data from the power grid, preprocess it, and generate structured and semantic data representations; S2: Construct an initial knowledge graph based on the structured and semantic data representation of the production; S3: Based on the initial knowledge graph, combined with the time-aware mechanism TGN, the compliance standards and patterns of the power grid operation environment change over time are identified. Through neural symbol fusion, rule-based logical reasoning is combined with a deep neural model to achieve compliance judgment under fuzzy conditions, thus optimizing the initial knowledge base into a dynamically evolving knowledge intelligence. S4: Based on knowledge-based intelligent agents, multimodal compliance analysis is performed by combining deep learning and symbolic reasoning to obtain analysis results; S5: Based on the analysis results and combined with natural language generation technology, automatically generate explanation reports for violations.

[0006] Furthermore, the acquisition of multi-source heterogeneous data from the power grid is specifically as follows: A unified access and acquisition mechanism for multi-source heterogeneous data from the power grid is established. For real-time monitoring data, standardized communication protocols are used to acquire real-time operating parameters, switch status, and alarm information time-series data of substations, lines, and generators. For equipment operation logs, a log parsing engine is used to extract fault records, maintenance information, and textual records of status changes from various equipment management systems. For operational image and video data, on-site operation photos, equipment inspection videos, and personnel operation recordings are acquired through on-site operation management platforms, inspection robots, and surveillance cameras. Operation and maintenance work ticket data is obtained from work order management systems and maintenance planning systems, acquiring structured forms containing key information such as operation content, executors, and time nodes. Procedure documents are collected from a document management system, acquiring unstructured texts of various safety procedures, operation manuals, and technical standards. Dispatch command records are obtained from the dispatch automation system, acquiring command logs containing information such as dispatch instructions, execution status, and response time.

[0007] Further preprocessing is performed as follows: After data collection, intelligent cleaning and preprocessing strategies are employed to ensure data quality. For structured data, statistical anomaly detection algorithms are used to identify and process missing, duplicate, and outlier values. Time series smoothing and interpolation algorithms are used to correct data discontinuities, and a unified data dictionary is established for field mapping and unit standardization. For semi-structured log data, regular expressions and NLP techniques are used for format parsing, extracting key business fields, and removing redundant information and system noise. For unstructured text data, a deep learning-based text cleaning model is used to remove irrelevant content, correct typos, and standardize terminology. Named entity recognition technology is used to automatically identify key entities such as device names, personnel information, and time and location. For multimedia data such as images and videos, computer vision preprocessing techniques are used for denoising, enhancement, and standardization, and key visual elements are extracted using target detection and scene recognition algorithms. For text-based data, a pre-trained language model is used for word embedding and sentence encoding to generate high-dimensional semantic vector representations. Semantic annotation and concept mapping are performed using domain-specific dictionaries and ontology libraries. For numerical monitoring data, statistical features, time-domain and frequency-domain features, and trend features are extracted using feature engineering methods, and standardized numerical representations are generated using normalization and standardization methods. For image and video data, visual feature vectors are extracted using convolutional neural networks and visual Transformer models, and multimodal fusion technology is used to align and map visual features with text descriptions. Finally, cross-modal representation learning methods are used to project different types of feature vectors onto a unified semantic space, forming a structured and semantic data representation that contains rich semantic information and is easy to compute. Furthermore, based on the structured and semantic data representations of the production, an initial knowledge graph is constructed, as follows: After the structured and semantic data representations are used, a pre-trained named entity recognition model is used to automatically identify and classify key entities in the power grid field; by constructing a power grid field ontology library and entity linking algorithm, the identified entities are matched with a standardized entity library to eliminate different representations of the same entity and ensure the consistency and accuracy of entity identification. Based on the identified entities, a relation extraction model is used to mine semantic relationships between entities and construct a relation network for power grid business. The relation extraction model adopts BERT-based relation extraction based on the attention mechanism to extract various types of binary relations from text data, such as equipment-belonging-substation, personnel-performing-operation, equipment-occurring-fault, and operation-following-procedure. At the same time, considering the characteristics of multimodal data, a cross-modal relation extraction method is adopted to establish semantic associations between images-text and videos-operation records. After completing entity recognition and relation extraction, a knowledge fusion model is used to integrate scattered knowledge fragments into an initial knowledge graph.

[0008] Furthermore, a knowledge fusion model is employed to integrate scattered knowledge fragments into an initial knowledge graph, specifically as follows: First, entity alignment and deduplication are performed, using entity embedding similarity calculation, string matching, and attribute comparison methods to identify duplicate entities. Clustering algorithms are then used to merge different entity representations of the same real-world object. Next, relation fusion and conflict resolution are performed. For contradictory or duplicate relation triples, timestamp comparison is used to resolve conflicts. Based on the extracted entities and relations, an initial knowledge graph is constructed using RDF triples or attribute graph models, where nodes represent power grid entities, edges represent relationships between entities, and the attributes of nodes and edges carry detailed semantic information.

[0009] Furthermore, the Time-Aware Graph Neural Network (TGN) is used to identify compliance standards and patterns in the power grid operating environment over time. Specifically, based on the constructed initial knowledge graph, the TGN is used to deeply model the time-varying characteristics of the power grid operating environment. By introducing a time encoding mechanism and a memory update module, the TGN can capture the dynamic change patterns of nodes and edges in the knowledge graph over time, identify seasonal operating patterns, equipment aging trends, and the temporal characteristics of evolving operating habits. The timestamp information of the power grid operating data is embedded into the graph structure to construct a time-enhanced knowledge representation Gt, forming a triplet structure. ; Where V is the set of nodes, containing all entities in the power grid; E t Let T be the set of edges at time t, representing the relationships between entities at a specific point in time; T is the set of timestamps, recording the specific times when all related events occurred, using Unix timestamps. When a new interaction event occurs, the memory state of the relevant node will be updated according to the new information: ; Among them, s i (t - ) represents the memory state vector of node i before time t; m i (t) is the message vector received by node i at time t, containing information from neighboring nodes; Δt is the time interval, representing the time difference between the last update and the current time; Memory(.) is the memory update function; s i (t) is the memory state vector of node i at time t; At each time step, a node collects information from its neighbors, including the neighbors' states, the characteristics of the connecting edges, and time information: ; Where Ni(t) is the set of neighbors of node i at time t; MSG(.) is the message function; e ij (t) represents the edge feature vector connecting nodes i and j, containing relation type, weight, and confidence level; AGG(.) is the aggregation function; s j (t - ) represents the memory state vector of node j before time t; s j (t) is the memory state vector of node j at time t; TimeEnc(t) is the time encoding vector at time t; By modeling historical interaction sequences using recurrent neural networks (RNNs), the temporal dependencies of entity states are learned. Simultaneously, an attention mechanism is used to dynamically adjust the importance weights of compliance standards in different time periods, automatically identifying compliance patterns that are strongly correlated with time. Through a continuous learning mechanism, the temporal patterns of the knowledge graph can be continuously updated based on newly added operational data, enabling adaptive perception and prediction of changes in the power grid operating environment.

[0010] Furthermore, rule-based logical reasoning is combined with deep neural models to achieve compliance judgments under fuzzy conditions. Specifically, a neural symbolic fusion reasoning framework is constructed, organically combining traditional rule-based symbolic reasoning with distributed representation learning of deep neural networks. At the symbolic level, explicit knowledge of power grid safety regulations, operating standards, and management systems is transformed into logical rules and constraints, and rule modeling is performed using first-order logic formal language. At the neural level, a graph neural network (GNN) is used to perform representation learning on the initial knowledge graph, generating low-dimensional embedding vectors of entities and relations. The nonlinear mapping capability of the neural network is used to handle noise, uncertainty, and fuzziness in the data. The two levels are coupled through a neural module network and a differentiable neural symbolic inferencer to achieve differentiability of symbolic reasoning and interpretability of neural computation. When faced with fuzzy conditions, it can integrate the deterministic reasoning of symbolic rules and the pattern recognition capability of neural networks to perform probabilistic compliance judgments and confidence assessments.

[0011] Furthermore, based on knowledge-based intelligent agents, multimodal compliance analysis is conducted by combining deep learning and symbolic reasoning to obtain the analysis results, as follows: Based on the constructed knowledge intelligent agent, the first step is to deeply fuse and semantically align the multi-source heterogeneous data generated during the power grid operation. Then, through cross-modal alignment technology, the features of different modalities are projected onto a unified multimodal representation space. The contrastive learning method is used to maximize the similarity between related modalities and minimize the similarity between unrelated modalities, thereby achieving effective fusion of multimodal information and ensuring semantic consistency. Based on multimodal feature fusion, compliance analysis is conducted by combining symbolic reasoning and neural reasoning. At the symbolic reasoning level, a formal rule base constructed according to power grid safety regulations, operating standards, and management systems is used to perform precise logical reasoning and deduction for explicit compliance conditions. At the neural reasoning level, graph neural networks are used to learn the representation of entity relationships in the knowledge agent, and complex association patterns are captured through multi-layer message propagation and attention mechanisms to handle fuzzy and uncertain compliance scenarios. After multimodal fusion and collaborative reasoning analysis, a comprehensive compliance analysis result is generated.

[0012] Furthermore, a comprehensive compliance analysis result is generated, as follows: First, the results of symbolic reasoning and neural reasoning are weighted and fused, and the weights are dynamically adjusted according to the reliability and completeness of different evidence sources to generate the final compliance probability distribution; Second, a Bayesian reasoning framework is used to model and quantify the uncertainty in the reasoning process, the parameter uncertainty is estimated by Monte Carlo sampling method, the model uncertainty is evaluated by ensemble model, and a confidence interval and credibility score are provided for each compliance judgment.

[0013] Furthermore, based on the analysis results and combined with natural language generation technology, an explanation report is automatically generated for the violation incident, as detailed below: Based on the multimodal compliance analysis results of the S4 stage, the detected violations are first subjected to in-depth structured analysis and key evidence extraction. Event graph construction technology is used to decompose violations into elements of time, location, personnel, equipment, operation, and rules. The complete chain and contextual information of the violation are traced through the entity relationship network in the knowledge graph. After completing the violation analysis and evidence extraction, an intelligent report generation architecture based on a combination of template-driven and deep learning is constructed. First, according to the standardization requirements of the power industry and the format specifications of regulatory departments, a multi-level report template system is designed, including a structured framework of violation summary templates, detailed analysis templates, and rectification suggestion templates. A sequence-to-sequence generation model based on the Transformer architecture is adopted, combined with a professional terminology dictionary and corpus in the power field for domain-adaptive fine-tuning, enabling the model to accurately understand and generate professional expressions that conform to power industry standards. Conditional text generation technology is introduced to dynamically select appropriate language styles and expressions based on the type of violation, severity, and equipment conditions involved.

[0014] A computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method steps described above.

[0015] The present invention has the following beneficial effects: 1. This invention uses knowledge graphs as its core to achieve deep integration of multi-source heterogeneous data from the power grid. Through systematic data preprocessing and unified semantic modeling, it not only breaks down the previously fragmented data of different types and data silos, but also constructs a unified and structured knowledge expression space, effectively improving data utilization and information coverage. On this basis, the knowledge graph establishes a clear semantic network for the complex relationships between equipment, personnel, operations, and procedures, providing a solid data foundation for subsequent complex compliance testing, risk tracing, and behavioral analysis, and significantly improving the comprehensiveness and accuracy of compliance monitoring. 2. This invention introduces a time-aware graph neural network and a neural symbolic fusion reasoning framework, which can not only dynamically capture the evolution of the power grid's operating status and behavior patterns over time, but also effectively balance the strictness of rule-based reasoning with the fault tolerance of deep models under fuzzy and uncertain conditions. The rule system of the symbolic layer ensures high security and interpretability, while the representation learning of the neural layer enhances the flexibility and intelligence of anomaly detection and compliance judgment. The fusion of the two enables the entire knowledge base to have self-learning, continuous evolution, and anomaly adaptation capabilities, making it capable of meeting the complex monitoring needs of frequent emergencies and information loss in actual power grid scenarios, and ensuring the system's high robustness and dynamic self-evolution capabilities. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: refer to Figure 1 In this embodiment, a knowledge graph-based method for monitoring compliance of multimodal data in power grids is provided, comprising the following steps: S1: Acquire multi-source heterogeneous data from the power grid, including SCADA real-time monitoring data, equipment operation logs, operation images and videos, maintenance operation tickets, procedure documents, and dispatch command records, and preprocess them to generate structured and semantic data representations; S2: Construct an initial knowledge graph based on the structured and semantic data representation of the production; S3: Based on the initial knowledge graph, combined with the time-aware mechanism TGN, the compliance standards and patterns of the power grid operation environment change over time are identified. Through neural symbol fusion, rule-based logical reasoning is combined with a deep neural model to achieve compliance judgment under fuzzy conditions, thus optimizing the initial knowledge base into a dynamically evolving knowledge intelligence. S4: Based on knowledge-based intelligent agents, multimodal compliance analysis is performed by combining deep learning and symbolic reasoning to obtain analysis results; S5: Based on the analysis results and combined with natural language generation technology, an explanation report is automatically generated for the violation event, such as: "The system detected that the disconnector switch QF2 has not been maintained according to the prescribed cycle. According to Article 5 of the Operation and Maintenance Procedures, there is a risk of the maintenance cycle being exceeded."

[0018] In this embodiment, the acquisition of multi-source heterogeneous data from the power grid is specifically as follows: A unified access and acquisition mechanism for multi-source heterogeneous data from the power grid is established. For real-time monitoring data, real-time operating parameters, switch status, and alarm information timing data of substations, lines, and generators are acquired through standardized communication protocols (such as IEC 61850 and DNP3.0). For equipment operation logs, a log parsing engine is used to extract fault records, maintenance information, and textual records of status changes from various equipment management systems. For operation image and video data, on-site operation photos, equipment inspection videos, and personnel operation recordings are acquired through on-site operation management platforms, inspection robots, and monitoring cameras. Operation and maintenance work ticket data is obtained from work order management systems and maintenance planning systems, using structured forms containing key information such as operation content, executors, and time nodes. Procedure documents are collected from various safety procedures, operation manuals, and technical standards in unstructured text through a document management system. Dispatch command records are obtained from the dispatch automation system, using command logs containing information such as dispatch instructions, execution status, and response time.

[0019] In this embodiment, the preprocessing is as follows: After data acquisition, intelligent cleaning and preprocessing strategies are adopted to ensure data quality. For structured data, statistical anomaly detection algorithms are used to identify and process missing values, duplicate values, and outliers. Time series smoothing and interpolation algorithms are used to repair data discontinuity issues, and a unified data dictionary is established for field mapping and unit standardization. For semi-structured log data, regular expressions and NLP techniques are used for format parsing, extracting key business fields, and removing redundant information and system noise. For unstructured text data, a deep learning-based text cleaning model is used to remove irrelevant content, correct typos, and standardize terminology. Named entity recognition technology is used to automatically identify key entities such as device names, personnel information, and time and location. For multimedia data such as images and videos, computer vision preprocessing techniques are used for denoising, enhancement, and standardization, and key visual elements are extracted through target detection and scene recognition algorithms. For text-based data, pre-trained language models (such as BERT and RoBERTa) are used for word embedding and sentence encoding to generate high-dimensional semantic vector representations. Simultaneously, domain-specific dictionaries and ontology libraries are used for semantic annotation and concept mapping. For numerical monitoring data, statistical features, time-domain and frequency-domain features, and trend features are extracted using feature engineering methods, and normalization and standardization methods are used to generate standardized numerical representations. For image and video data, convolutional neural networks and visual Transformer models are used to extract visual feature vectors, and multimodal fusion technology is used to align and map visual features with text descriptions. Finally, cross-modal representation learning methods are used to project different types of feature vectors onto a unified semantic space, forming a structured and semantic data representation that contains rich semantic information and is easy to compute. In this embodiment, an initial knowledge graph is constructed based on the structured and semantic data representation of the production data, as follows: After the structured and semantic data representation is established, a pre-trained named entity recognition model is used to automatically identify and classify key entities in the power grid domain; considering the complexity of power grid terminology and equipment names, a BERT-CRF sequence labeling model is used, combined with a power industry-specific dictionary and labeled corpus for domain-adaptive fine-tuning, to achieve accurate identification of equipment entities such as substations, transmission lines, generator sets, switching equipment, and protection devices; simultaneously, personnel entities (such as operators, maintenance personnel, etc.) are identified. The system identifies multiple types of entities, including dispatchers, time-based entities (operation time, maintenance cycle, alarm time), geographical entities (substation location, line route), and status entities (operation status, alarm level, operation type). By constructing a power grid ontology library and entity linking algorithms, the identified entities are matched against a standardized entity library to eliminate different representations of the same entity, ensuring consistency and accuracy of entity identification. Furthermore, for image and video data, target detection and scene understanding technologies are used to identify visual entities, such as equipment appearance, work tools, and safety equipment, and these are then associated and mapped with corresponding text entities. Based on the identified entities, a relation extraction model is used to mine semantic relationships between entities and construct a relation network for power grid business. The relation extraction model adopts BERT-based relation extraction based on an attention mechanism to extract various types of binary relations from text data, such as equipment-belongs-substation, personnel-execute-operation, equipment-occurs-fault, and operation-follows-procedure. Considering the characteristics of time-series data, special attention is paid to the extraction of time-dependent relations, such as temporal relations such as "equipment A-state changes to-abnormal at time T" and "operation B-duration-30 minutes". At the same time, considering the characteristics of multimodal data, a cross-modal relation extraction method is adopted to establish semantic associations between images-text and videos-operation records, such as cross-modal relations such as "on-site image-display-equipment status" and "operation video-record-operation process". After completing entity recognition and relation extraction, a knowledge fusion model is used to integrate scattered knowledge fragments into an initial knowledge graph.

[0020] In this embodiment, a knowledge fusion model is used to integrate scattered knowledge fragments into an initial knowledge graph, as follows: First, entity alignment and deduplication are performed. Duplicate entities are identified using entity embedding similarity calculation, string matching, and attribute comparison methods. Different entity representations of the same real-world object are merged using a clustering algorithm. Then, relation fusion and conflict resolution are performed. For contradictory or duplicate relation triples, timestamp comparison is used to resolve conflicts. Based on the extracted entities and relations, an initial knowledge graph is constructed using RDF triples or attribute graph models, where nodes represent power grid entities, edges represent relationships between entities, and the attributes of nodes and edges carry detailed semantic information.

[0021] In this embodiment, the Time-Aware Graph Neural Network (TGN) is used to identify compliance standards and patterns in the power grid operating environment over time. Specifically, based on the constructed initial knowledge graph, the TGN is used to deeply model the time-varying characteristics of the power grid operating environment. By introducing a time encoding mechanism and a memory update module, the TGN can capture the dynamic change patterns of nodes and edges in the knowledge graph over time, identify seasonal operating patterns, equipment aging trends, and the temporal characteristics of evolving operating habits. The timestamp information of the power grid operating data is embedded into the graph structure to construct a time-enhanced knowledge representation Gt, forming a triplet structure (subject-relationship-object-time). ; Where V is the set of nodes, containing all entities in the power grid, such as transformers, circuit breakers, relay protection devices, maintenance personnel, dispatchers, etc.; E t Let T be the set of edges at time t, representing the relationships between entities at a specific point in time, such as "device A - connected to - bus B" or "person C - operating - switch D"; T is the set of timestamps, recording the specific time points when all related events occurred, using Unix timestamps. When a new interaction event occurs, the memory state of the relevant node will be updated according to the new information: ; Among them, s i (t - ) represents the memory state vector of node i before time t; m i (t) is the message vector received by node i at time t, containing information from neighboring nodes; Δt is the time interval, representing the time difference between the last update and the current time; Memory(.) is the memory update function; s i (t) is the memory state vector of node i at time t; At each time step, a node collects information from its neighbors, including the neighbors' states, the characteristics of the connecting edges, and time information: ; Where Ni(t) is the set of neighbors of node i at time t; MSG(.) is the message function, which is a multilayer perceptron (MLP) with the same output dimension as the node embedding dimension; e ij (t) represents the edge feature vector connecting nodes i and j, containing relation type, weight, and confidence level; AGG(.) is the aggregation function; s j (t - ) represents the memory state vector of node j before time t; s j (t) is the memory state vector of node j at time t; TimeEnc(t) is the time encoding vector at time t; By modeling historical interaction sequences using recurrent neural networks (RNNs), the temporal dependencies of entity states are learned. Simultaneously, an attention mechanism is used to dynamically adjust the importance weights of compliance standards in different time periods, automatically identifying time-sensitive compliance patterns such as "inspection frequency requirements during high summer loads," "duty regulations during holidays," and "safety measures during equipment maintenance seasons." Through a continuous learning mechanism, the temporal patterns of the knowledge graph can be continuously updated based on newly added operational data, enabling adaptive perception and prediction of changes in the power grid operating environment.

[0022] In this embodiment, rule-based logical reasoning is combined with a deep neural model to achieve compliance judgment under fuzzy conditions. Specifically, a neural symbolic fusion reasoning framework is constructed, organically combining traditional rule-based symbolic reasoning with distributed representation learning of deep neural networks. At the symbolic level, explicit knowledge of power grid safety regulations, operating standards, and management systems is transformed into logical rules and constraints. Rule modeling is performed using a first-order logic formal language, supporting precise logical reasoning and deduction. At the neural level, a graph neural network (GNN) is used to perform representation learning on the initial knowledge graph, generating low-dimensional embedding vectors of entities and relations. The nonlinear mapping capability of the neural network is used to handle noise, uncertainty, and fuzziness in the data. The two levels are coupled through a neural module network and a differentiable neural symbolic inferencer to achieve differentiability of symbolic reasoning and interpretability of neural computation. When faced with fuzzy conditions such as "device status signal missing but surrounding environment abnormal" or "incomplete operation record but on-site image shows violation," the system can comprehensively combine the deterministic reasoning of symbolic rules and the pattern recognition capability of neural networks to perform probabilistic compliance judgment and confidence assessment.

[0023] In this embodiment, multimodal compliance analysis is performed based on a knowledge-based intelligent agent, combined with deep learning and symbolic reasoning, to obtain the analysis results, as follows: Based on the constructed knowledge-based intelligent agent, the system first performs deep fusion and semantic alignment of multi-source heterogeneous data generated during power grid operation. For text modal data (such as operation logs, procedure documents, and dispatch instructions), pre-trained language models (BERT, RoBERTa) are used to extract semantic features, and key elements such as equipment names, operation types, and time information in the text are mapped to a unified semantic space through entity linking technology in the knowledge graph. For image modal data (on-site operation photos and equipment inspection images), convolutional neural networks and visual Transformers are used to extract visual features, and object detection algorithms are used to identify key visual elements such as equipment status, personnel behavior, and safety equipment wearing status. For time-series numerical data (SCADA monitoring data and sensor readings), time-domain and frequency-domain features are extracted using time-series convolutional networks and LSTM networks to identify equipment operation trends and abnormal patterns. Through cross-modal alignment technology, the features of different modalities are projected onto a unified multimodal representation space. Contrastive learning methods are used to maximize the similarity between related modalities and minimize the similarity between unrelated modalities, thereby achieving effective fusion of multimodal information and ensuring semantic consistency. Based on multimodal feature fusion, compliance analysis is conducted using a combination of symbolic reasoning and neural reasoning. At the symbolic reasoning level, a formal rule base constructed from power grid safety regulations, operating standards, and management systems is used to perform precise logical reasoning and deduction for explicit compliance conditions. For example, deterministic rules such as "grounding wires must be connected during high-voltage equipment maintenance" and "personnel entering high-voltage areas must wear insulating equipment" are rigorously verified through the symbolic reasoning engine. At the neural reasoning level, graph neural networks are used to learn representations of entity relationships in knowledge agents. Complex association patterns are captured through multi-layer message propagation and attention mechanisms to handle fuzzy and uncertain compliance scenarios, such as "abnormal equipment status signals but no clear fault indication" and "incomplete operation records but on-site images showing standardized operation." After multimodal fusion and collaborative reasoning analysis, a comprehensive compliance analysis result is generated.

[0024] In this embodiment, a comprehensive compliance analysis result is generated as follows: First, the results of symbolic reasoning and neural reasoning are weighted and fused, and the weights are dynamically adjusted according to the reliability and completeness of different evidence sources to generate the final compliance probability distribution; Second, a Bayesian reasoning framework is used to model and quantify the uncertainty in the reasoning process, the parameter uncertainty is estimated by the Monte Carlo sampling method, the model uncertainty is evaluated by the ensemble model, and a confidence interval and credibility score are provided for each compliance judgment.

[0025] In this embodiment, based on the analysis results and combined with natural language generation technology, an explanation report is automatically generated for the violation event, as detailed below: Based on the multimodal compliance analysis results of the S4 phase, the detected violations are first subjected to in-depth structured analysis and key evidence extraction. Event graph construction technology is used to decompose violations into elements of time, location, personnel, equipment, operation, and rules. The complete chain and contextual information of the violation are traced through the entity relationship network in the knowledge graph. For textual evidence, named entity recognition and relation extraction technologies are used to extract key information from operation logs and work order records, such as "Violation time: October 15, 2023, 14:30" and "Equipment involved: 110kV transformer T1". Structured elements such as "Operator: Zhang Moumou" and "Violation of rule: Article 4.2.3 of the 'Electric Power Safety Work Regulations'" are used. For image evidence, computer vision technology is used to identify on-site violations, such as key visual features like "not wearing a safety helmet," "illegally crossing a safety fence," and "not grounding the equipment during maintenance." Target detection and scene understanding algorithms are used to locate the specific location of the violation in the image. For numerical evidence, anomaly detection algorithms are used to identify monitoring data that deviates from the normal range, such as quantitative indicators like "equipment temperature exceeds the standard by 50%" and "insulation resistance value is lower than the safety threshold." The system establishes an evidence credibility assessment mechanism, assigning credibility weights to each piece of evidence based on factors such as the authority of the data source, consistency over time, and the degree of mutual corroboration among multiple sources, ensuring the accuracy and reliability of subsequent report generation. After completing the analysis of violation events and evidence extraction, an intelligent report generation architecture based on a combination of template-driven and deep learning is constructed. First, according to the standardization requirements of the power industry and the format specifications of regulatory departments, a multi-level report template system is designed, including a structured framework of violation summary templates, detailed analysis templates, and rectification suggestion templates. A sequence-to-sequence generation model with a Transformer architecture (such as T5 and BART) is adopted, and domain-adaptive fine-tuning is performed in conjunction with a professional terminology dictionary and corpus in the power field, enabling the model to accurately understand and generate professional expressions that conform to power industry standards. Conditional text generation technology is introduced to dynamically select appropriate language styles and expressions based on the type, severity, and equipment conditions involved in the violation. For example, a serious warning tone is used for safety violations, while an objective descriptive tone is used for operational violations.

[0026] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0027] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0028] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0029] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0030] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A knowledge graph-based method for monitoring compliance of multimodal data in power grids, characterized in that, Includes the following steps: S1: Acquire multi-source heterogeneous data from the power grid, preprocess it, and generate structured and semantic data representations; S2: Construct an initial knowledge graph based on the structured and semantic data representation of the production; S3: Based on the initial knowledge graph, combined with the time-aware mechanism TGN, the compliance standards and patterns of the power grid operation environment change over time are identified. Through neural symbol fusion, rule-based logical reasoning is combined with a deep neural model to achieve compliance judgment under fuzzy conditions, thus optimizing the initial knowledge base into a dynamically evolving knowledge intelligence. S4: Based on knowledge-based intelligent agents, multimodal compliance analysis is performed by combining deep learning and symbolic reasoning to obtain analysis results; S5: Based on the analysis results and combined with natural language generation technology, an explanation report is automatically generated for the violation. The rule-based logical reasoning combined with deep neural models enables compliance judgment under fuzzy conditions, specifically as follows: A neural symbolic fusion reasoning framework is constructed, organically combining traditional rule-based symbolic reasoning with distributed representation learning of deep neural networks; at the symbolic level, explicit knowledge of power grid safety regulations, operating standards, and management systems is transformed into logical rules and constraints, and rule modeling is performed using first-order logic formal language; at the neural level, a graph neural network (GNN) is used to perform representation learning on the initial knowledge graph, generating low-dimensional embedding vectors of entities and relations, and using the nonlinear mapping capability of the neural network to handle noise, uncertainty, and fuzziness in the data; The two levels are coupled through a neural module network and a differentiable neural symbolic inferencer to achieve differentiability of symbolic reasoning and interpretability of neural computation. When faced with fuzzy conditions, it can combine the deterministic reasoning of symbolic rules with the pattern recognition capability of neural networks to perform probabilistic compliance judgments and confidence assessments.

2. The knowledge graph-based power grid multimodal data compliance monitoring method according to claim 1, characterized in that, The acquisition of multi-source heterogeneous data from the power grid is specifically as follows: A unified access and acquisition mechanism for multi-source heterogeneous data from the power grid is established. For real-time monitoring data, standardized communication protocols are used to acquire real-time operating parameters, switch status, and alarm information time-series data of substations, lines, and generators. For equipment operation logs, a log parsing engine is used to extract fault records, maintenance information, and textual records of status changes from various equipment management systems. For operation image and video data, on-site operation photos, equipment inspection videos, and personnel operation recordings are acquired through on-site operation management platforms, inspection robots, and surveillance cameras. Maintenance operation ticket data is obtained from work order management systems and maintenance planning systems, acquiring structured forms containing key information such as operation content, executors, and time nodes. Procedure documents are collected from a document management system, acquiring unstructured texts of various safety procedures, operation manuals, and technical standards. Dispatch command records are obtained from the dispatch automation system, acquiring command logs containing information such as dispatch instructions, execution status, and response time.

3. The knowledge graph-based power grid multimodal data compliance monitoring method according to claim 2, characterized in that, The preprocessing is specifically as follows: After data acquisition, intelligent cleaning and preprocessing strategies are adopted to ensure data quality. For structured data, statistical anomaly detection algorithms are used to identify and process missing values, duplicate values, and outliers. Time series smoothing and interpolation algorithms are used to repair data discontinuities, and a unified data dictionary is established for field mapping and unit standardization. For semi-structured log data, regular expressions and NLP techniques are used for format parsing, extracting key business fields, and removing redundant information and system noise. For unstructured text data, a deep learning-based text cleaning model is used to remove irrelevant content, correct typos, and standardize terminology. Named entity recognition technology is used to automatically identify key entities such as device names, personnel information, and time and location. For multimedia data such as images and videos, computer vision preprocessing techniques are used for noise reduction and enhancement. The process involves standardization and extraction of key visual elements using object detection and scene recognition algorithms. For textual data, pre-trained language models are used for word embedding and sentence encoding to generate high-dimensional semantic vector representations. Semantic annotation and concept mapping are performed using domain-specific dictionaries and ontology libraries. For numerical monitoring data, statistical features, time-domain and frequency-domain features, and trend features are extracted using feature engineering methods, and standardized numerical representations are generated using normalization and standardization methods. For image and video data, convolutional neural networks and visual Transformer models are used to extract visual feature vectors, and multimodal fusion technology is used to align and map visual features with textual descriptions. Finally, cross-modal representation learning methods are used to project different types of feature vectors onto a unified semantic space, forming a structured and semantic data representation that contains rich semantic information and is easy to compute.

4. The knowledge graph-based power grid multimodal data compliance monitoring method according to claim 1, characterized in that, The initial knowledge graph is constructed based on the structured and semantic data representations produced, as follows: After the structured and semantic data representations are established, a pre-trained named entity recognition model is used to automatically identify and classify key entities in the power grid domain; by constructing a power grid domain ontology library and entity linking algorithm, the identified entities are matched with a standardized entity library to eliminate different representations of the same entity and ensure the consistency and accuracy of entity identification; based on the identified entities, a relation extraction model is used to mine the semantic relationships between entities and construct a relation network for power grid business; the relation extraction model adopts BERT-based relation extraction based on the attention mechanism to extract various types of binary relations from text data, such as equipment-belongs-substation, personnel-execute-operation, equipment-occur-fault, and operation-follow-procedure; at the same time, considering the characteristics of multimodal data, a cross-modal relation extraction method is adopted to establish semantic associations between images-text and videos-operation records; after completing entity recognition and relation extraction, a knowledge fusion model is used to integrate scattered knowledge fragments into an initial knowledge graph.

5. The knowledge graph-based power grid multimodal data compliance monitoring method according to claim 4, characterized in that, The knowledge fusion model integrates scattered knowledge fragments into an initial knowledge graph, as follows: First, entity alignment and deduplication are performed. Duplicate entities are identified using entity embedding similarity calculation, string matching, and attribute comparison methods. Different entity representations of the same real-world object are merged using a clustering algorithm. Then, relation fusion and conflict resolution are performed. For contradictory or duplicate relation triples, timestamp comparison is used to resolve conflicts. Based on the extracted entities and relations, an initial knowledge graph is constructed using RDF triples or attribute graph models, where nodes represent power grid entities, edges represent relationships between entities, and the attributes of nodes and edges carry detailed semantic information.

6. The knowledge graph-based power grid multimodal data compliance monitoring method according to claim 1, characterized in that, The method of using a time-aware graph neural network (TGN) to identify compliance standards and patterns in the power grid operating environment over time is as follows: Based on the constructed initial knowledge graph, a time-aware graph neural network (TGN) is used to deeply model the time-varying characteristics of the power grid operating environment. By introducing a time encoding mechanism and a memory update module, TGN can capture the dynamic change patterns of nodes and edges in the knowledge graph over time, identify seasonal operating patterns, equipment aging trends, and the temporal characteristics of evolving operating habits. It embeds the timestamp information of the power grid operating data into the graph structure, constructing a time-enhanced knowledge representation Gt, forming a triplet structure. ； Where V is the set of nodes, containing all entities in the power grid; E t Let T be the set of edges at time t, representing the relationships between entities at a specific point in time; T is the set of timestamps, recording the specific times when all related events occurred, using Unix timestamps. When a new interaction event occurs, the memory state of the relevant node will be updated according to the new information: ； Among them, s i (t - ) represents the memory state vector of node i before time t; m i (t) is the message vector received by node i at time t, containing information from neighboring nodes; Δt is the time interval, representing the time difference between the last update and the current time; Memory(.) is the memory update function; s i (t) is the memory state vector of node i at time t; At each time step, a node collects information from its neighbors, including the neighbors' states, the characteristics of the connecting edges, and time information: ； Where Ni(t) is the set of neighbors of node i at time t; MSG(.) is the message function; e ij (t) represents the edge feature vector connecting nodes i and j, containing relation type, weight, and confidence level; AGG(.) is the aggregation function; s j (t - ) represents the memory state vector of node j before time t; s j (t) is the memory state vector of node j at time t; TimeEnc(t) is the time encoding vector at time t; By modeling historical interaction sequences using recurrent neural networks (RNNs), the temporal dependencies of entity states are learned. Simultaneously, an attention mechanism is used to dynamically adjust the importance weights of compliance standards in different time periods, automatically identifying compliance patterns that are strongly correlated with time. Through a continuous learning mechanism, the temporal patterns of the knowledge graph can be continuously updated based on newly added operational data, enabling adaptive perception and prediction of changes in the power grid operating environment.

7. The knowledge graph-based power grid multimodal data compliance monitoring method according to claim 1, characterized in that, The knowledge-based intelligent agent, combined with deep learning and symbolic reasoning, performs multimodal compliance analysis to obtain the following results: Based on the constructed knowledge intelligent agent, the first step is to deeply fuse and semantically align the multi-source heterogeneous data generated during the power grid operation. Then, through cross-modal alignment technology, the features of different modalities are projected onto a unified multimodal representation space. The contrastive learning method is used to maximize the similarity between related modalities and minimize the similarity between unrelated modalities, thereby achieving effective fusion of multimodal information and ensuring semantic consistency. Based on multimodal feature fusion, compliance analysis is conducted by combining symbolic reasoning and neural reasoning. At the symbolic reasoning level, a formal rule base constructed according to power grid safety regulations, operating standards, and management systems is used to perform precise logical reasoning and deduction for explicit compliance conditions. At the neural reasoning level, graph neural networks are used to learn the representation of entity relationships in the knowledge agent, and complex association patterns are captured through multi-layer message propagation and attention mechanisms to handle fuzzy and uncertain compliance scenarios. After multimodal fusion and collaborative reasoning analysis, a comprehensive compliance analysis result is generated.

8. The knowledge graph-based power grid multimodal data compliance monitoring method according to claim 1, characterized in that, The process of generating comprehensive compliance analysis results is as follows: First, the results of symbolic reasoning and neural reasoning are weighted and fused, and the weights are dynamically adjusted according to the reliability and completeness of different evidence sources to generate the final compliance probability distribution. Second, a Bayesian reasoning framework is used to model and quantify the uncertainty in the reasoning process, the parameter uncertainty is estimated by Monte Carlo sampling, the model uncertainty is evaluated by ensemble model, and a confidence interval and credibility score are provided for each compliance judgment.

9. The knowledge graph-based power grid multimodal data compliance monitoring method according to claim 1, characterized in that, Based on the analysis results and combined with natural language generation technology, an explanation report is automatically generated for the violation incident, as detailed below: Based on the multimodal compliance analysis results of the S4 stage, the detected violations are first analyzed in a deep structure and key evidence is extracted. Event graph construction technology is used to decompose the violations into elements of time, location, personnel, equipment, operation, and rules. The complete chain and contextual information of the violation are traced through the entity relationship network in the knowledge graph. After completing the analysis of the violations and the extraction of evidence, an intelligent report generation architecture based on a combination of template-driven and deep learning is constructed. First, according to the standardization requirements of the power industry and the format specifications of regulatory authorities, a multi-level report template system is designed, including a structured framework of violation summary templates, detailed analysis templates, and rectification suggestion templates. A sequence-to-sequence generation model based on the Transformer architecture is adopted, and domain-adaptive fine-tuning is performed by combining a professional terminology dictionary and corpus in the power industry, enabling the model to accurately understand and generate professional expressions that conform to power industry standards. Conditional text generation technology is introduced to dynamically select appropriate language styles and expressions based on the type of violation, severity, and equipment conditions involved.

Images