Power grid operation and maintenance sensitive operation identification method based on multi-modal fusion and related equipment

Abstract

The application relates to the technical field of electric digital data processing, in particular to a power grid operation and maintenance sensitive operation identification method based on multi-modal fusion and related equipment, wherein the method comprises the following steps: collecting multi-modal data such as operation log texts, monitoring video images and operation audios, extracting semantic, visual and acoustic features after preprocessing, realizing feature fusion by using a multi-modal fusion model containing a feature alignment layer and an attention mechanism layer, and then identifying sensitive operations by a classifier and generating early warnings. The method breaks through the limitation of traditional single data source, solves the problems of low artificial auditing efficiency, high missed detection rate and insufficient single modal identification capability, realizes automatic intelligent auditing and real-time early warning of operation and maintenance operations, and improves the efficiency and safety of power grid operation and maintenance management.

Description

Technical Field

[0001] This application relates to the field of electrical digital data processing technology, and in particular to a method and related equipment for identifying sensitive operations in power grid operation and maintenance based on multimodal fusion. Background Technology

[0002] In power system operation and maintenance management, traditional operation and maintenance review relies heavily on manual verification of log files, monitoring videos, and other data. This manual review method is not only inefficient and unable to meet the massive data processing demands brought about by the development of smart grids, but also limited by subjective perception, making it easy to overlook violations in complex operation scenarios and creating hidden dangers for the safe operation of the power grid.

[0003] With the digital upgrade of smart grids, the amount of multimodal data generated by operation and maintenance work, such as text logs, monitoring videos, and operation audio, is growing exponentially, urgently requiring automation technology to improve the efficiency and accuracy of auditing. However, most existing operation and maintenance operation recognition technologies are based on analysis of a single data source (such as log text or monitoring video). This approach has a fundamental flaw: single-modal data cannot cover the multidimensional characteristics of operation and maintenance operations, and it is difficult to capture the implicit relationships between different information carriers such as text descriptions, operational behaviors, and voice commands. In particular, the recognition accuracy is seriously insufficient for complex and sensitive operations involving the combination of text and images (such as the linkage between log records and monitoring screens when equipment malfunctions).

[0004] Therefore, based on the above problems, the existing technology still needs to be improved. Summary of the Invention

[0005] The purpose of this application is to provide a method and related equipment for identifying sensitive operations in power grid operation and maintenance based on multimodal fusion, aiming to solve the problems of missed detection and misjudgment of violations caused by low efficiency of manual review and insufficient single-modal recognition capabilities.

[0006] The purpose of this application is to provide a method for identifying sensitive operations in power grid operation and maintenance based on multimodal fusion, including:

[0007] Collect multimodal data in power grid operation and maintenance scenarios, including operation log text, monitoring video images, and operation audio;

[0008] The multimodal data is preprocessed, including text cleaning, video frame extraction, and audio noise reduction;

[0009] Semantic features are extracted from the operation log text using a natural language processing model;

[0010] Visual features are extracted from the surveillance video images using a convolutional neural network.

[0011] Acoustic features are extracted from the operational audio using a recurrent neural network;

[0012] The semantic features, visual features, and acoustic features are input into a multimodal fusion model for feature fusion. The multimodal fusion model includes a feature alignment layer and an attention mechanism layer.

[0013] Based on the fused features, a classifier is used to identify whether sensitive operations exist.

[0014] In response to the identification of sensitive operations, an early warning message is generated and pushed to the operation and maintenance management platform.

[0015] By adopting the above technical solution, multimodal data such as operation log text, monitoring video images, and operation audio in power grid operation and maintenance scenarios can be effectively integrated. Preprocessing steps such as text cleaning, video frame extraction, and audio noise reduction improve data quality. Natural language processing models, convolutional neural networks, and recurrent neural networks are used to extract semantic, visual, and acoustic features from different modalities, respectively. A multimodal fusion model including feature alignment layers and attention mechanisms achieves deep correlation and fusion of cross-modal features. Then, a classifier accurately identifies sensitive operations and generates early warning information, which is pushed to the operation and maintenance management platform. This approach overcomes the limitations of traditional single-data source identification, comprehensively capturing operation and maintenance details through collaborative analysis of multimodal data. It effectively improves the accuracy and robustness of sensitive operation identification, solves the problems of low efficiency and high false negative rates in manual review, and realizes automated intelligent review and real-time early warning of operation and maintenance operations. This significantly improves the efficiency and security of power grid operation and maintenance management, providing strong technical support for the safe operation of the power grid.

[0016] In one possible implementation of this application, the step of collecting multimodal data in a power grid operation and maintenance scenario includes:

[0017] In substation areas with high electromagnetic interference, an electromagnetic interference-resistant sensor network and signal isolation technology are used to collect the operation log text, monitoring video images and operation audio data, and signal noise reduction processing is used to suppress power frequency interference and transient noise generated by switching operations.

[0018] For the monitoring video images collected by the outdoor inspection robot in rainy and snowy weather, video motion compensation technology is used to eliminate image blurring, and multispectral image fusion technology is used to enhance the recognition ability of abnormal conditions of equipment joints.

[0019] In multi-device collaborative operation scenarios, the device communication protocol data stream is parsed, the opening and closing timing characteristics of circuit breakers and disconnect switches are extracted, and the timestamps are aligned with the operation log text.

[0020] By adopting the above technical solutions, in substation areas with high electromagnetic interference, anti-electromagnetic interference sensor networks and signal isolation technologies, combined with signal noise reduction processing, can effectively suppress power frequency interference and transient noise generated by switching operations, ensuring the acquisition quality of operation log text, monitoring video images, and operation audio data. For monitoring video images collected by outdoor inspection robots in rainy and snowy weather, video motion compensation technology is used to eliminate image blurring caused by robot movement. At the same time, multispectral image fusion technology is used to enhance the identification capability of abnormal equipment joint states, improving image recognition performance in complex environments. In multi-device collaborative operation scenarios, the opening and closing timing characteristics of circuit breakers and disconnectors are extracted by parsing the equipment communication protocol data stream and timestamped with the operation log text to achieve time-series correlation analysis of multi-source data. This allows for comprehensive and accurate capture of key features of multimodal data in power grid operation and maintenance scenarios, effectively improving the reliability and stability of sensitive operation identification under complex operating conditions, and providing more accurate data support for the safety management of power grid operation and maintenance.

[0021] In one possible implementation of this application, the step of inputting the semantic features, visual features, and acoustic features into a multimodal fusion model for feature fusion includes:

[0022] A knowledge graph of power grid equipment containing equipment models, operating specifications, and historical fault cases is constructed, and the features of the equipment knowledge graph are aligned with the semantic features across modalities through a graph attention network.

[0023] Based on the importance level of the equipment and the scope of its operational impact, corresponding feature fusion weights are assigned to transformers, GIS combined electrical appliances, and circuit breakers.

[0024] A dynamic routing algorithm is used to fuse temporal operation features. The time interval of continuous operation, the device start-up and shutdown sequence, and the trend of operation parameter changes are encoded into attention vectors, which are then gating and fused with the visual features.

[0025] By adopting the above technical solutions, a power grid equipment knowledge graph containing equipment models, operating specifications, and historical fault cases can be constructed. A graph attention network is used to achieve cross-modal alignment of equipment knowledge graph features and semantic features, enabling multimodal data to form an associated mapping in a unified semantic space. Differentiated feature fusion weights are assigned to key equipment such as transformers, GIS switchgear, and circuit breakers based on equipment importance level and operational impact range, prioritizing feature extraction and analysis of high-risk equipment and improving the targeting of sensitive operation identification. A dynamic routing algorithm encodes temporal features such as continuous operation time intervals, equipment start-up and shutdown sequences, and operating parameter change trends into attention vectors, and performs gating fusion with visual features. This captures the temporal logic and dynamic associations in operation and maintenance, achieving deep fusion and intelligent weighting of multimodal features. This effectively improves the completeness of feature representation and the accuracy of semantic understanding, enabling the model to more accurately identify sensitive operations in power grid operation and maintenance, reducing false positives and false negatives, and enhancing the ability to identify complex operation scenarios and key equipment.

[0026] In one possible implementation of this application, the step of identifying the presence of sensitive operations based on the fused features using a classifier includes:

[0027] Establish a knowledge base for power grid operation and maintenance specifications, transform prohibited and mandatory operation specifications into rule vectors, and perform joint reasoning with the aforementioned multimodal features;

[0028] For predefined typical violations, a state machine model based on temporal logic is constructed, and the compliance of the operation sequence is verified temporally through sequence analysis;

[0029] When training the classifier, the multimodal feature distance between normal and illegal operations is calculated in the semantic space. The ability to identify sensitive operations is enhanced by maximizing the separability between features.

[0030] By adopting the above technical solutions, prohibited and mandatory operation specifications in the power grid operation and maintenance standard knowledge base can be transformed into rule vectors, which can then be used for joint reasoning with multimodal features. This allows the classifier to closely integrate industry standards when identifying sensitive operations, improving the accuracy and compliance of reasoning. For typical violations, a state machine model based on temporal logic is constructed. Sequence analysis is used to verify the compliance of operation sequences temporally, effectively capturing the temporal order and logical relationships during the operation process and avoiding misjudgments caused by single feature analysis. When training the classifier, calculating the multimodal feature distance between normal and violation operations in the semantic space and maximizing the separability between features enhances the classifier's learning ability for sensitive operation features, enabling it to more accurately distinguish between normal and violation operations. This significantly improves the accuracy and reliability of sensitive operation identification, effectively reducing missed and false judgments, and providing stronger technical support for power grid operation and maintenance safety.

[0031] In one possible implementation of this application, the early warning information generation step includes:

[0032] Based on the degree of operational risk, sensitive operations are divided into different priorities. For operations of different priorities, the corresponding emergency response system is triggered and the control loops of related equipment are locked.

[0033] Generate a visual early warning report containing equipment spatial location information, mark the location of the violation in the virtual scene, and match and display historical handling solutions;

[0034] A multi-channel redundant push mechanism is adopted to ensure that critical early warning information can still be sent when the main communication link is interrupted.

[0035] By adopting the above technical solutions, sensitive operations can be classified into different priorities based on their operational risk levels. For operations of different priorities, corresponding emergency response systems are triggered, and relevant equipment control loops are linked and locked, enabling rapid intervention and safe blocking of high-risk operations, effectively preventing the escalation of accidents. A visualized early warning report containing equipment spatial location information is generated, accurately marking the location of violations in a virtual scenario and displaying historical handling plans, providing maintenance personnel with intuitive fault location references and handling strategy support, significantly shortening fault response time. A multi-channel redundant push mechanism is adopted, using backup channels such as BeiDou satellite short messages to ensure reliable transmission of critical early warning information even when main communication links such as 5G are interrupted, guaranteeing the stability and reliability of the early warning system in complex communication environments. This constructs a three-dimensional early warning system covering risk classification, visualized positioning, and multi-channel protection, comprehensively improving the efficiency and safety of emergency response to sensitive operations in power grid maintenance.

[0036] In one possible implementation of this application, the method further includes:

[0037] An operation and maintenance simulation environment is constructed, and the multimodal fusion model is optimized through a reinforcement learning mechanism with the goal of reducing the false judgment rate and shortening the early warning response time.

[0038] When an undefined violation type is identified, suspicious samples are automatically labeled and the training dataset is updated, triggering the model's iterative learning mechanism;

[0039] Regularly generate model performance evaluation reports, and separately calculate the accuracy and false negative rate for key operation types.

[0040] By adopting the above technical solutions, an operation and maintenance simulation environment can be constructed, and a multimodal fusion model can be optimized in a targeted manner using reinforcement learning mechanisms. This aims to continuously improve model performance by reducing the false positive rate and shortening the early warning response time, making it more adaptable to complex operation and maintenance scenarios. When an undefined type of violation is identified, suspicious samples are automatically marked and the training dataset is updated, triggering the model's iterative learning mechanism. This endows the system with autonomous learning and evolution capabilities, effectively solving the problem of identifying new types of violations. Regular model performance evaluation reports are generated, and the accuracy and false negative rates for key operation types are statistically analyzed separately, providing data support for continuous model optimization and ensuring high reliability and stability of the system in actual operation. This forms a closed-loop optimization system of "simulation training - autonomous learning - performance monitoring," comprehensively improving the accuracy, adaptability, and long-term reliability of identifying sensitive operations in power grid operation and maintenance.

[0041] In one possible implementation of this application, the method further includes:

[0042] Establish an association index between the multimodal features and the equipment ledger, and perform cross-modal matching of the equipment name in the operation log text, the equipment number in the monitoring video image, the operation instructions in the operation audio, and the asset information in the equipment lifecycle management system;

[0043] The multimodal features of sensitive operations are stored on the blockchain, and the stored content includes feature vectors, timestamps, and operator identity information to ensure the immutability of operation traceability.

[0044] In equipment maintenance scenarios, transfer learning can be used to transfer historical sensitive operational features to the recognition model of new equipment, reducing the cold start problem of the model in the early stage of new equipment commissioning.

[0045] By adopting the above technical solutions, a correlation index between multimodal features and equipment ledgers can be established. This allows for cross-modal matching of equipment names in operation log texts, equipment numbers in monitoring video images, operation commands in operation audio, and asset information in the equipment lifecycle management system. This achieves precise correlation between operation and maintenance data and equipment assets, providing data support for equipment status assessment and lifecycle management. Furthermore, multimodal features of sensitive operations are stored on the blockchain, with the stored content including feature vectors, timestamps, and operator identity information, ensuring the immutability of operation traceability and providing a reliable basis for power system security audits and liability determination. In equipment maintenance scenarios, transfer learning is used to transfer historical sensitive operation features to the recognition model of new equipment, reducing the cold start problem of models in the early stages of new equipment commissioning, shortening the model training cycle, and improving the accuracy and reliability of sensitive operation recognition for new equipment. This comprehensively enhances the intelligence level and security assurance capabilities of power grid operation and maintenance management.

[0046] The second objective of this application is to provide a power grid operation and maintenance sensitive operation identification system based on multimodal fusion, the system comprising:

[0047] Multimodal data acquisition module: Acquires multimodal data in power grid operation and maintenance scenarios, including operation log text, monitoring video images, and operation audio;

[0048] Multimodal data preprocessing module: preprocesses the multimodal data, including text cleaning, video frame extraction, and audio noise reduction;

[0049] Semantic feature extraction module: Extracts semantic features from the operation log text using a natural language processing model;

[0050] Visual feature extraction module: Extracts visual features from the surveillance video images using a convolutional neural network;

[0051] Acoustic feature extraction module: Extracts acoustic features from the operational audio using a recurrent neural network;

[0052] Feature fusion processing module: Inputs the semantic features, visual features and acoustic features into a multimodal fusion model for feature fusion. The multimodal fusion model includes a feature alignment layer and an attention mechanism layer.

[0053] Sensitive Operation Identification Module: Based on the fused features, a classifier is used to identify whether a sensitive operation exists;

[0054] Warning information generation module: In response to the identification of sensitive operations, it generates warning information and pushes it to the operation and maintenance management platform.

[0055] By adopting the above technical solution, multimodal data such as operation log text, monitoring video images, and operation audio in power grid operation and maintenance scenarios can be effectively integrated. Preprocessing steps such as text cleaning, video frame extraction, and audio noise reduction improve data quality. Natural language processing models, convolutional neural networks, and recurrent neural networks are used to extract semantic, visual, and acoustic features from different modalities, respectively. A multimodal fusion model including feature alignment layers and attention mechanisms achieves deep correlation and fusion of cross-modal features. Then, a classifier accurately identifies sensitive operations and generates early warning information, which is pushed to the operation and maintenance management platform. This approach overcomes the limitations of traditional single-data source identification, comprehensively capturing operation and maintenance details through collaborative analysis of multimodal data. It effectively improves the accuracy and robustness of sensitive operation identification, solves the problems of low efficiency and high false negative rates in manual review, and realizes automated intelligent review and real-time early warning of operation and maintenance operations. This significantly improves the efficiency and security of power grid operation and maintenance management, providing strong technical support for the safe operation of the power grid.

[0056] The third objective of this application is to provide a power grid operation and maintenance sensitive operation identification device based on multimodal fusion, the device comprising:

[0057] The system includes a memory and a processor, wherein the memory stores a computer program that can be loaded and executed by the processor to identify the power grid operation and maintenance sensitive operation based on multimodal fusion.

[0058] The fourth objective of this application is to provide a storage medium.

[0059] The fourth objective of this application is achieved through the following technical solution:

[0060] A storage medium storing a computer program capable of being loaded by a processor and executed by the aforementioned method for identifying sensitive operations in power grid operation and maintenance based on multimodal fusion.

[0061] In summary, this application includes at least one of the following beneficial technical effects:

[0062] 1. This system effectively integrates multimodal data from power grid operation and maintenance scenarios, including operation log text, monitoring video images, and operation audio. Preprocessing steps such as text cleaning, video frame extraction, and audio noise reduction improve data quality. Natural language processing models, convolutional neural networks, and recurrent neural networks extract semantic, visual, and acoustic features from different modalities. A multimodal fusion model, including feature alignment and attention mechanisms, achieves deep correlation and fusion of cross-modal features. A classifier then accurately identifies sensitive operations and generates early warning information, which is pushed to the operation and maintenance management platform. This overcomes the limitations of traditional single-data source identification, comprehensively capturing operation and maintenance details through collaborative analysis of multimodal data. It effectively improves the accuracy and robustness of sensitive operation identification, solving the problems of low efficiency and high false negative rates in manual review. It achieves automated intelligent review and real-time early warning of operation and maintenance operations, significantly improving the efficiency and security of power grid operation and maintenance management, and providing strong technical support for the safe operation of the power grid.

[0063] 2. It can establish a correlation index between multimodal features and equipment ledgers, performing cross-modal matching of equipment names in operation log texts, equipment numbers in monitoring video images, operation instructions in operation audio, and asset information in the equipment lifecycle management system. This achieves accurate association between operation and maintenance data and equipment assets, providing data support for equipment status assessment and lifecycle management. It also uses blockchain to store multimodal features of sensitive operations, including feature vectors, timestamps, and operator identity information, ensuring the immutability of operation traceability and providing a reliable basis for power system safety audits and liability determination. In equipment maintenance scenarios, it uses transfer learning to transfer historical sensitive operation features to the recognition model of new equipment, reducing the cold start problem of models in the early stages of new equipment commissioning, shortening the model training cycle, and improving the accuracy and reliability of sensitive operation recognition for new equipment. This comprehensively enhances the intelligence level and safety assurance capabilities of power grid operation and maintenance management. Attached Figure Description

[0064] Figure 1 This is a flowchart illustrating the power grid operation and maintenance sensitive operation identification method based on multimodal fusion provided in the embodiments of this application;

[0065] Figure 2 This is a virtual structure diagram of the power grid operation and maintenance sensitive operation identification system based on multimodal fusion provided in the embodiments of this application. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0067] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0068] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0069] This application provides a method for identifying sensitive operations in power grid operation and maintenance based on multimodal fusion, referring to... Figure 1 The main process of the method is described as follows:

[0070] S1: Collect multimodal data in power grid operation and maintenance scenarios, including operation log text, monitoring video images, and operation audio;

[0071] The multimodal data acquisition is achieved through a sensor network deployed on equipment such as substations and inspection robots: operation log text is generated in real time by the power monitoring system, monitoring video images are captured by high-definition cameras to show equipment operation scenes, and operation audio is captured by microphones to record on-site operation commands. The acquisition scope covers typical operation and maintenance scenarios such as equipment maintenance, switching operations, and fault handling, ensuring data integrity.

[0072] S2: Preprocess the multimodal data, including text cleaning, video frame extraction, and audio noise reduction;

[0073] The preprocessing steps include: text cleaning by removing redundant characters and erroneous encodings from log text; video frame extraction by extracting keyframes from surveillance video at a fixed frame rate to reduce data volume; and audio denoising using algorithms such as wavelet transform to eliminate environmental noise and improve speech clarity. These preprocessing steps aim to improve the accuracy of subsequent feature extraction.

[0074] S3: Extract semantic features from the operation log text using a natural language processing model;

[0075] In this process, pre-trained language models such as BERT are used to encode the operation log text, parse key information such as operation instructions, device names, and timestamps, and generate semantic vectors representing the operation intentions, thereby achieving the structured transformation of text information.

[0076] S4: Extract visual features from the surveillance video images using a convolutional neural network;

[0077] Among them, convolutional neural network architectures such as ResNet are used to process monitoring video frames to identify visual features such as the appearance status of equipment (e.g., the opening and closing position of circuit breakers) and operation gestures. Multi-layer feature extraction networks are used to capture image representations of abnormal actions or violations of equipment.

[0078] S5: Extract acoustic features from the operational audio using a recurrent neural network;

[0079] The audio signal is converted into a Mel spectrogram and then input into a recurrent neural network such as LSTM to extract the temporal features of the voice command, identify the compliance of the operation command (such as whether it contains prohibited operation terms), and generate an acoustic vector representing the semantics of the audio.

[0080] S6: Input the semantic features, visual features and acoustic features into a multimodal fusion model for feature fusion. The multimodal fusion model includes a feature alignment layer and an attention mechanism layer.

[0081] The multimodal fusion model first maps semantic, visual, and acoustic features to a unified semantic space through a feature alignment layer to eliminate modal differences; then, it uses an attention mechanism layer to automatically assign feature weights based on dimensions such as equipment importance and operating scenario to enhance the fusion effect of key information (such as transformer operation-related features).

[0082] S7: Based on the fused features, a classifier is used to identify whether there are sensitive operations;

[0083] The classifier uses a support vector machine (SVM) or a deep neural network. It learns the feature boundaries between normal and illegal operations through training, classifies and distinguishes the fused feature vectors, and outputs the identification results of sensitive operations (such as illegal types such as "pulling the disconnect switch under load").

[0084] S8: In response to the identification of sensitive operations, generate early warning information and push it to the operation and maintenance management platform.

[0085] When a sensitive operation is detected, the system generates an early warning message that includes the operation type, risk level, time and location according to preset rules. This message is then pushed to the operation and maintenance management platform in real time via wired / wireless communication networks, providing operation and maintenance personnel with immediate action guidelines.

[0086] Specifically, in some possible embodiments, the step of collecting multimodal data in power grid operation and maintenance scenarios includes:

[0087] In substation areas with high electromagnetic interference, an electromagnetic interference-resistant sensor network and signal isolation technology are used to collect the operation log text, monitoring video images and operation audio data, and signal noise reduction processing is used to suppress power frequency interference and transient noise generated by switching operations.

[0088] For the monitoring video images collected by the outdoor inspection robot in rainy and snowy weather, video motion compensation technology is used to eliminate image blurring, and multispectral image fusion technology is used to enhance the recognition ability of abnormal conditions of equipment joints.

[0089] In multi-device collaborative operation scenarios, the device communication protocol data stream is parsed, the opening and closing timing characteristics of circuit breakers and disconnect switches are extracted, and the timestamps are aligned with the operation log text.

[0090] In the process of collecting multimodal data in power grid operation and maintenance scenarios, in substation areas with high electromagnetic interference, anti-electromagnetic interference sensor networks and signal isolation technologies are deployed to collect operation log text, monitoring video images, and operation audio data. Signal noise reduction processing is used to suppress power frequency interference and transient noise generated by switch operations. For monitoring video images collected by outdoor inspection robots in rainy and snowy weather, video motion compensation technology is used to eliminate image blurring caused by robot movement. At the same time, multispectral image fusion technology is used to enhance the ability to identify abnormal states of equipment joints. In multi-device collaborative operation scenarios, the equipment communication protocol data stream is parsed to extract the opening and closing timing characteristics of circuit breakers and disconnectors, and these are timestamped and aligned with the operation log text.

[0091] In power transmission line drone inspection scenarios, infrared thermal imagers and lidar are used to simultaneously collect conductor temperature distribution and spatial location data, combined with visible light images to identify anomalies such as line galloping and hanging foreign objects. In distributed power access scenarios, data such as power generation and power quality are obtained by analyzing inverter communication protocols, and multimodal correlation is performed with the equipment appearance status in inspection videos. In deeply extended embodiments, fiber optic sensor networks can be used to achieve opto-isolated signal transmission in high electromagnetic interference environments, combined with adaptive filtering algorithms to dynamically adjust noise reduction parameters. For multi-device collaborative operation, a Petri net-based time-series logic model can be constructed to formally verify the compliance of operation sequences, further improving the accuracy of time-series feature analysis.

[0092] Specifically, in some possible embodiments, the step of inputting the semantic features, visual features, and acoustic features into a multimodal fusion model for feature fusion includes:

[0093] A knowledge graph of power grid equipment containing equipment models, operating specifications, and historical fault cases is constructed, and the features of the equipment knowledge graph are aligned with the semantic features across modalities through a graph attention network.

[0094] Based on the importance level of the equipment and the scope of its operational impact, corresponding feature fusion weights are assigned to transformers, GIS combined electrical appliances, and circuit breakers.

[0095] A dynamic routing algorithm is used to fuse temporal operation features. The time interval of continuous operation, the device start-up and shutdown sequence, and the trend of operation parameter changes are encoded into attention vectors, which are then gating and fused with the visual features.

[0096] When inputting semantic, visual, and acoustic features into a multimodal fusion model for feature fusion, a power grid equipment knowledge graph containing equipment models, operating specifications, and historical fault cases is first constructed. Graph Attention Network (GAT) is used to calculate the correlation between entities and semantic features in the knowledge graph, and weights are automatically assigned through an attention mechanism to achieve cross-modal alignment. For high-value equipment such as transformers and GIS combined electrical appliances, the weight coefficients are dynamically adjusted during feature fusion based on their importance level in the power grid (e.g., N-1 verification results) and the scope of operational impact (e.g., the size of the outage area), strengthening the expression of key equipment features. A dynamic routing algorithm (e.g., the dynamic routing mechanism in Capsule Networks) is used to process time-series operational features, encoding the time interval of continuous operations, equipment start-up and shutdown sequence, and operational parameter change trends into attention vectors. Visual features are selectively fused through gating units (e.g., the forget gate structure in LSTM) to suppress redundant information interference.

[0097] This approach incorporates operational environment features (such as environmental parameters like temperature and humidity) and equipment status features (such as real-time monitoring data like oil temperature and vibration levels). A multimodal autoencoder aligns the three-dimensional features of the environment, equipment, and operation. For renewable energy integration scenarios, a knowledge graph encompassing the characteristics of wind, solar, and energy storage equipment is constructed, integrating power fluctuation characteristics, meteorological forecast data, and other operational features with maintenance characteristics. In a deeply extended embodiment, a hierarchical graph attention network can be designed, with the lower layer focusing on knowledge associations at the equipment component level and the higher layer extracting system-level operational patterns. Causal reasoning is employed to enhance temporal feature fusion, using a Do-calculus method to identify causal relationships in operational sequences, filtering statistical correlation interference, and improving the model's generalization ability to unseen operational patterns.

[0098] Specifically, in some possible embodiments, the step of identifying the presence of sensitive operations using a classifier based on the fused features includes:

[0099] Establish a knowledge base for power grid operation and maintenance specifications, transform prohibited and mandatory operation specifications into rule vectors, and perform joint reasoning with the aforementioned multimodal features;

[0100] For predefined typical violations, a state machine model based on temporal logic is constructed, and the compliance of the operation sequence is verified temporally through sequence analysis;

[0101] When training the classifier, the multimodal feature distance between normal and illegal operations is calculated in the semantic space. The ability to identify sensitive operations is enhanced by maximizing the separability between features.

[0102] In the process of identifying sensitive operations using a classifier based on fused features, a knowledge base of power grid operation and maintenance specifications, including regulations such as the "Power Safety Work Regulations," is first established. Prohibited operations such as "prohibiting disconnecting switches under load" and mandatory operations such as "verification of voltage is required before operation" are transformed into binary rule vectors, which are then used in conjunction with multimodal features through tensor product operations for joint reasoning. For typical violations such as "misoperation of main transformer circuit breakers," a finite state machine (FSM) is used to construct a temporal logic model, defining triplet rules of "equipment state - operation action - state transition." A sliding window is used to verify the compliance of the operation sequence. When training the classifier, a triplet loss function is used to calculate the cosine distance between the multimodal features of normal and non-compliant operations in the semantic space. An optimizer maximizes the inter-class distance and minimizes the intra-class distance to enhance the feature discrimination capability of sensitive operations.

[0103] Infrared thermal imaging data from drone inspections is introduced to perform cross-modal joint reasoning by combining abnormal equipment temperature characteristics with "overload" descriptions in operation log texts. In microgrid operation and maintenance scenarios, a standardized knowledge base containing distributed power generation switching rules is constructed, combining the power change curve characteristics of photovoltaic inverter start-up and shutdown operations with a time-series state machine model. In a deeply extended embodiment, a federated learning framework is used to construct an operation standard knowledge base across substations, protecting the data privacy of each site through differential privacy while achieving collaborative updates of rule vectors. A causal reasoning model is introduced to identify the causal relationship chain of "misoperation-fault" based on the causal graph structure of the operation sequence, rather than relying solely on statistical correlation, thereby improving the generalization ability to identify new types of illegal operations.

[0104] Specifically, in some possible embodiments, the warning information generation step includes:

[0105] Based on the degree of operational risk, sensitive operations are divided into different priorities. For operations of different priorities, the corresponding emergency response system is triggered and the control loops of related equipment are locked.

[0106] Generate a visual early warning report containing equipment spatial location information, mark the location of the violation in the virtual scene, and match and display historical handling solutions;

[0107] A multi-channel redundant push mechanism is adopted to ensure that critical early warning information can still be sent when the main communication link is interrupted.

[0108] Specifically, an operational risk assessment model is constructed using the Analytic Hierarchy Process (AHP), classifying sensitive operations into three priorities: emergency (e.g., power outage under load), important (e.g., non-sequential operation), and general (e.g., operation timeout). These priorities trigger red, orange, and yellow emergency response systems, respectively. Simultaneously, programmable logic controllers (PLCs) are used to interlock the control loops of relevant equipment. A visualized early warning report, including equipment GPS coordinates and 3D spatial location, is generated based on a Geographic Information System (GIS). Equipment violating operational rules is marked in a digital twin scenario, and historical similar cases are matched using a knowledge graph to determine appropriate handling solutions. A three-channel redundant push mechanism using 5G, BeiDou satellite short message service, and power line carrier communication (PLC) is employed, automatically switching to the backup channel when the main communication link is interrupted, ensuring reliable transmission of critical early warning information under extreme conditions.

[0109] Develop an augmented reality (AR)-based early warning interaction system to overlay virtual handling solutions onto field equipment; for UHVDC transmission scenarios, construct an early warning knowledge base including converter valve status parameters to achieve precise equipment-level early warning; in multi-energy collaborative operation and maintenance scenarios, design a multi-agent-based early warning decision system to generate optimal handling solutions through a negotiation mechanism. In a deeply extended embodiment, a risk propagation model based on Bayesian networks can be constructed to dynamically deduce the spatiotemporal evolution path of operational risks; a federated learning framework can be adopted to achieve cross-regional early warning knowledge sharing, improving the ability to identify new types of violations; and digital watermarking technology can be used to verify early warning information against counterfeiting, ensuring the integrity and credibility of information transmission.

[0110] Specifically, in some possible embodiments, the method further includes:

[0111] An operation and maintenance simulation environment is constructed, and the multimodal fusion model is optimized through a reinforcement learning mechanism with the goal of reducing the false judgment rate and shortening the early warning response time.

[0112] When an undefined violation type is identified, suspicious samples are automatically labeled and the training dataset is updated, triggering the model's iterative learning mechanism;

[0113] Regularly generate model performance evaluation reports, and separately calculate the accuracy and false negative rate for key operation types.

[0114] This involves constructing an operation and maintenance simulation environment that includes a digital twin of the power system, training the agent using the Proximal Policy Optimization (PPO) algorithm, and using the minimization of the false positive rate and early warning response time as the reward function. Adversarial training is used to enhance the robustness of the model. When an undefined combination of illegal keywords is detected in the operation log text (such as "not tested for power" and "closed" appearing simultaneously), or when the equipment status in the video image does not match the operation instructions, it is automatically marked as a suspicious sample. After expert review, the training set is updated and incremental learning is triggered. A model performance evaluation report is generated weekly. For key operation types such as "switching operation" and "equipment maintenance", the accuracy, recall, and F1 score based on the confusion matrix are calculated, and ROC curves are plotted to evaluate the classification performance of the model.

[0115] Develop a virtual reality (VR)-based operation and maintenance training system to collect operational data in real time by simulating erroneous operation scenarios and expand abnormal samples using generative adversarial networks (GANs). For distributed power supply access scenarios, construct a knowledge graph containing microgrid control strategies to achieve semantic understanding and risk assessment of new operation and maintenance operations. In a deeply extended embodiment, a model adaptation mechanism based on meta-learning (MAML) can be designed to enable the model to converge quickly with a small number of samples when encountering new operation types. Do-calculus analysis is used to identify confounding variables in model decision-making, and counterfactual reasoning is used to optimize feature fusion strategies, thereby improving the model's generalization ability to complex operation scenarios.

[0116] Specifically, in some possible embodiments, the method further includes:

[0117] Establish an association index between the multimodal features and the equipment ledger, and perform cross-modal matching of the equipment name in the operation log text, the equipment number in the monitoring video image, the operation instructions in the operation audio, and the asset information in the equipment lifecycle management system;

[0118] The multimodal features of sensitive operations are stored on the blockchain, and the stored content includes feature vectors, timestamps, and operator identity information to ensure the immutability of operation traceability.

[0119] In equipment maintenance scenarios, transfer learning can be used to transfer historical sensitive operational features to the recognition model of new equipment, reducing the cold start problem of the model in the early stage of new equipment commissioning.

[0120] Specifically, an inverted index of multimodal features and equipment ledgers is established using Elasticsearch. Equipment names in operation log text are extracted using Named Entity Recognition (NER) and spatiotemporally aligned with equipment numbers identified by OCR in monitoring video images and operation commands extracted by speech recognition in operation audio. Then, cross-modal matching is performed with asset information (such as model, commissioning time, and maintenance records) in the equipment lifecycle management system. The Hyperledger Fabric blockchain platform is used to store the multimodal features of sensitive operations. Semantic feature vectors, visual feature vectors, acoustic feature vectors, and information such as timestamps, operator IDs, and equipment IDs are hashed and uploaded to the blockchain. Smart contracts are used to verify the immutability of the stored data. In the scenario of new equipment commissioning and maintenance, transfer learning is used to transfer the sensitive operation features of similar historical equipment (such as the multimodal features of transformer opening and closing operations) from the source domain model to the target domain model. By freezing the pre-trained layer parameters and fine-tuning the classification layer, the cold start problem of the model caused by insufficient samples in the early stage of new equipment commissioning is reduced.

[0121] The association index between multimodal features and equipment ledgers is applied to wind power equipment operation and maintenance scenarios, performing cross-modal matching between wind turbine vibration audio, blade infrared images, and gearbox models and blade serial numbers in the wind turbine ledger. When storing multimodal features of sensitive operations in photovoltaic power plants on the blockchain, inverter operating parameters and meteorological data are simultaneously stored, enabling multi-dimensional association for operation traceability and fault analysis. In energy storage power plant equipment maintenance scenarios, historical features of lithium battery charging and discharging operations are transferred to the identification model of new energy storage battery clusters through transfer learning, combined with federated learning technology to protect the data privacy of each site. In a deeply extended embodiment, a knowledge graph-based multimodal feature-equipment ledger association network can be constructed, using graph convolutional networks (GCNs) to mine implicit associations in equipment operation history; zero-knowledge proof technology is used to hide sensitive feature details in blockchain storage while ensuring the integrity verification of the stored data; a meta-learning-based transfer strategy is designed to quickly adapt to the equipment-specific operating modes using a small number of samples during the cold start phase of a new equipment model, improving the generalization ability of transfer learning.

[0122] Another embodiment of this application provides a power grid operation and maintenance sensitive operation identification system based on multimodal fusion, wherein, see reference Figure 2 A power grid operation and maintenance sensitive operation identification system based on multimodal fusion includes:

[0123] Multimodal data acquisition module 100: Acquires multimodal data in power grid operation and maintenance scenarios, including operation log text, monitoring video images, and operation audio;

[0124] Multimodal data preprocessing module 200: preprocesses the multimodal data, including text cleaning, video frame extraction, and audio noise reduction;

[0125] Semantic feature extraction module 300: Extracts semantic features from the operation log text using a natural language processing model;

[0126] Visual feature extraction module 400: Extracts visual features from the surveillance video images using a convolutional neural network;

[0127] Acoustic feature extraction module 500: Extracts acoustic features from the operational audio through a recurrent neural network;

[0128] Feature fusion processing module 600: Inputs the semantic features, visual features and acoustic features into a multimodal fusion model for feature fusion, wherein the multimodal fusion model includes a feature alignment layer and an attention mechanism layer;

[0129] Sensitive Operation Identification Module 700: Based on the fused features, it identifies whether a sensitive operation exists through a classifier;

[0130] Warning Information Generation Module 800: In response to the identification of sensitive operations, it generates warning information and pushes it to the operation and maintenance management platform.

[0131] The power grid operation and maintenance sensitive operation identification system based on multimodal fusion provided in this embodiment can realize the steps of the aforementioned embodiments due to the functions of each module and the logical connection between them. Therefore, it can achieve the same technical effect as the aforementioned embodiments. For the principle analysis, please refer to the relevant description of the steps of the aforementioned power grid operation and maintenance sensitive operation identification method based on multimodal fusion, which will not be repeated here.

[0132] This application embodiment also provides a power grid operation and maintenance sensitive operation identification device based on multimodal fusion, including a memory and a processor. The memory stores a computer program that can be loaded by the processor and executed by the above-mentioned power grid operation and maintenance sensitive operation identification method based on multimodal fusion.

[0133] This application embodiment also provides a storage medium storing a computer program that can be loaded by a processor and executed by the above-described method for identifying sensitive operations in power grid operation and maintenance based on multimodal fusion.

[0134] The storage medium provided in this embodiment can achieve the same technical effect as the aforementioned embodiments because the computer program therein, after being loaded and run on the processor, will implement the various steps of the aforementioned embodiments. For the principle analysis, please refer to the relevant description of the aforementioned method steps, which will not be repeated here.

[0135] The storage medium includes, for example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0137] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0138] Furthermore, features defined by the terms "first" and "second" may explicitly or implicitly include at least one of those features. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., and unless otherwise explicitly specified, is used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0139] Therefore, any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0140] The embodiments described in this specific implementation are preferred embodiments of this application and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for identifying sensitive operations in power grid operation and maintenance based on multimodal fusion, characterized in that, include: Collect multimodal data in power grid operation and maintenance scenarios, including operation log text, monitoring video images, and operation audio; The steps for collecting multimodal data in power grid operation and maintenance scenarios include: in substation areas with high electromagnetic interference, using electromagnetic interference-resistant sensor networks and signal isolation technology to collect the operation log text, monitoring video images, and operation audio data, and suppressing power frequency interference and transient noise generated by switch operations through signal noise reduction processing; for the monitoring video images collected by outdoor inspection robots in rainy and snowy weather, using video motion compensation technology to eliminate image blurring, and simultaneously enhancing the identification capability of abnormal equipment joint states through multispectral image fusion technology; in multi-device collaborative operation scenarios, parsing the equipment communication protocol data stream, extracting the opening and closing timing characteristics of circuit breakers and disconnectors, and aligning them with the timestamps of the operation log text; The multimodal data is preprocessed, including text cleaning, video frame extraction, and audio noise reduction; Semantic features are extracted from the operation log text using a natural language processing model; Visual features are extracted from the surveillance video images using a convolutional neural network. Acoustic features are extracted from the operational audio using a recurrent neural network; The semantic, visual, and acoustic features are input into a multimodal fusion model for feature fusion. The multimodal fusion model includes a feature alignment layer and an attention mechanism layer. Specific steps include: constructing a power grid equipment knowledge graph containing equipment models, operating specifications, and historical fault cases; aligning the features of the equipment knowledge graph with the semantic features across modalities using a graph attention network; assigning corresponding feature fusion weights to transformers, GIS combined electrical appliances, and circuit breakers based on equipment importance levels and operational impact ranges; and fusing temporal operational features using a dynamic routing algorithm, encoding the time intervals of continuous operations, equipment start-up and shutdown sequences, and operational parameter change trends into attention vectors, and performing gated fusion with the visual features. Based on the fused features, a classifier is used to identify whether sensitive operations exist. In response to the identification of sensitive operations, an early warning message is generated and pushed to the operation and maintenance management platform; Establish an association index between the multimodal features and the equipment ledger, and perform cross-modal matching of the equipment name in the operation log text, the equipment number in the monitoring video image, the operation instructions in the operation audio, and the asset information in the equipment lifecycle management system.

2. The method for identifying sensitive operations in power grid operation and maintenance based on multimodal fusion according to claim 1, characterized in that, The step of identifying the presence of sensitive operations using a classifier based on the fused features includes: Establish a knowledge base for power grid operation and maintenance specifications, transform prohibited and mandatory operation specifications into rule vectors, and perform joint reasoning with the aforementioned multimodal features; For predefined typical violations, a state machine model based on temporal logic is constructed, and the compliance of the operation sequence is verified temporally through sequence analysis; When training the classifier, the multimodal feature distance between normal and illegal operations is calculated in the semantic space. The ability to identify sensitive operations is enhanced by maximizing the separability between features.

3. The method for identifying sensitive operations in power grid operation and maintenance based on multimodal fusion according to claim 1, characterized in that, The steps for generating the early warning information include: Based on the degree of operational risk, sensitive operations are divided into different priorities. For operations of different priorities, the corresponding emergency response system is triggered and the control loops of related equipment are locked. Generate a visual early warning report containing equipment spatial location information, mark the location of the violation in the virtual scene, and match and display historical handling solutions; A multi-channel redundant push mechanism is adopted to ensure that critical early warning information can still be sent when the main communication link is interrupted.

4. The method for identifying sensitive operations in power grid operation and maintenance based on multimodal fusion according to claim 1, characterized in that, The method further includes: An operation and maintenance simulation environment is constructed, and the multimodal fusion model is optimized through a reinforcement learning mechanism with the goal of reducing the false judgment rate and shortening the early warning response time. When an undefined violation type is identified, suspicious samples are automatically labeled and the training dataset is updated, triggering the model's iterative learning mechanism; Regularly generate model performance evaluation reports, and separately calculate the accuracy and false negative rate for key operation types.

5. The method for identifying sensitive operations in power grid operation and maintenance based on multimodal fusion according to claim 1, characterized in that, The method further includes: The multimodal features of sensitive operations are stored on the blockchain, and the stored content includes feature vectors, timestamps, and operator identity information to ensure the immutability of operation traceability. In equipment maintenance scenarios, transfer learning can be used to transfer historical sensitive operational features to the recognition model of new equipment, reducing the cold start problem of the model in the early stage of new equipment commissioning.

6. A power grid operation and maintenance sensitive operation identification system based on multimodal fusion, characterized in that, include: Multimodal data acquisition module: Acquires multimodal data in power grid operation and maintenance scenarios, including operation log text, monitoring video images, and operation audio; The steps for collecting multimodal data in power grid operation and maintenance scenarios include: in substation areas with high electromagnetic interference, using electromagnetic interference-resistant sensor networks and signal isolation technology to collect the operation log text, monitoring video images, and operation audio data, and suppressing power frequency interference and transient noise generated by switch operations through signal noise reduction processing; for the monitoring video images collected by outdoor inspection robots in rainy and snowy weather, using video motion compensation technology to eliminate image blurring, and simultaneously enhancing the identification capability of abnormal equipment joint states through multispectral image fusion technology; in multi-device collaborative operation scenarios, parsing the equipment communication protocol data stream, extracting the opening and closing timing characteristics of circuit breakers and disconnectors, and aligning them with the timestamps of the operation log text; Multimodal data preprocessing module: preprocesses the multimodal data, including text cleaning, video frame extraction, and audio noise reduction; Semantic feature extraction module: Extracts semantic features from the operation log text using a natural language processing model; Visual feature extraction module: Extracts visual features from the surveillance video images using a convolutional neural network; Acoustic feature extraction module: Extracts acoustic features from the operational audio using a recurrent neural network; Feature fusion processing module: Inputs the semantic features, visual features, and acoustic features into a multimodal fusion model for feature fusion. The multimodal fusion model includes a feature alignment layer and an attention mechanism layer. Specific steps include: constructing a power grid equipment knowledge graph containing equipment models, operating specifications, and historical fault cases; aligning the features of the equipment knowledge graph with the semantic features across modalities using a graph attention network; assigning corresponding feature fusion weights to transformers, GIS combined electrical appliances, and circuit breakers based on equipment importance level and operational impact range; and fusing temporal operation features using a dynamic routing algorithm, encoding the time interval of continuous operations, equipment start-up and shutdown sequence, and operational parameter change trends into attention vectors, and performing gated fusion with the visual features. Sensitive Operation Identification Module: Based on the fused features, a classifier is used to identify whether a sensitive operation exists; Warning information generation module: In response to the identification of sensitive operations, it generates warning information and pushes it to the operation and maintenance management platform; it establishes an association index between the multimodal features and the equipment ledger, and performs cross-modal matching of the equipment name in the operation log text, the equipment number in the monitoring video image, the operation command in the operation audio, and the asset information in the equipment life cycle management system.

7. A power grid operation and maintenance sensitive operation identification device based on multimodal fusion, characterized in that, include: The memory and processor, wherein the memory stores a computer program that can be loaded by the processor and executed as any of the above-described methods for identifying sensitive power grid operation and maintenance operations based on multimodal fusion, as described in claims 1-5.

8. A storage medium, characterized in that, The computer program is stored and can be loaded by a processor to execute any of the above-described methods for identifying sensitive power grid operation and maintenance operations based on multimodal fusion, as described in claims 1-5.

Images