Power equipment comprehensive online monitoring method and system based on multi-modal artificial intelligence

Abstract

The application provides a power equipment comprehensive online monitoring method and system based on a multi-modal artificial intelligence. The method comprises the following steps: synchronously collecting sample data sets of power equipment through a mobile inspection platform, and generating a power equipment number for identifying the power equipment to which the sample belongs; performing multi-modal fusion recognition on an edge AI inference platform, outputting a state abnormality category of the equipment to which the current sample belongs and a behavior abnormality category recognized in the current image, and generating a standard event description five tuple; constructing a causal graph based on the standard event description five tuple, calculating the causal scores between nodes through a structure connection enhancement and a spatial distance attenuation mechanism, generating an abnormal causal graph with causal scores and a causal path set of each state abnormality node, and mapping the causal graph and the causal path set of each state abnormality node to a three-dimensional digital twin model. The application provides a complete solution path for constructing a smart substation operation and maintenance system.

Description

Technical Field

[0001] This invention belongs to the field of integrated online monitoring of power equipment, and particularly relates to a method and system for integrated online monitoring of power equipment based on multimodal artificial intelligence. Background Technology

[0002] With the continuous expansion of the power grid and the rapid increase in the number of primary equipment, substation condition awareness and inspection maintenance face multiple challenges in terms of accuracy, efficiency, and stability. Traditional substation inspection methods mostly rely on periodic manual checks, supplemented by fixed-point video surveillance and some equipment status sensors, which cannot achieve panoramic, continuous, and high-frequency condition acquisition and dynamic anomaly identification of complex electrical systems. In recent years, although some sites have introduced robotic inspection systems, drone image acquisition systems, thermal imaging analysis devices, and local AI analysis modules, these systems generally suffer from bottlenecks such as modality separation, platform fragmentation, static deployment, and uninterpretable events. Specifically, video images and equipment sensor data lack a spatiotemporal alignment mechanism, behavioral operations and equipment status lack event-related annotations, and recognition algorithms for different modalities are deployed in independent systems without linkage, resulting in a high misjudgment rate for anomaly identification and making it difficult to support practical operation and maintenance in complex scenarios.

[0003] Meanwhile, existing AI algorithms largely rely on centralized training and lack edge deployment capabilities. The accuracy of recognition models drops significantly when on-site equipment structures or lighting conditions change, exhibiting a lack of adaptability and structural stability. More critically, even when some anomalies are identified, the system often fails to provide a clear path to the anomaly's formation and its underlying causes, thus failing to complete the logical loop from "seeing the anomaly" to "understanding the anomaly." The image monitoring, status recognition, digital twin display, and AI model invocation functions in existing platforms are often fragmented, lacking a unified graph-driven mechanism for information aggregation, path construction, and semantic visual interaction.

[0004] Therefore, how to build a comprehensive intelligent monitoring system based on on-site data collection and with integrated capabilities of identification, reasoning, and visualization has become a key technical issue that needs to be addressed in the high-reliability operation and maintenance system of smart substations. Summary of the Invention

[0005] The purpose of this invention is to propose a comprehensive online monitoring method and system for power equipment based on multimodal artificial intelligence, thereby solving the above-mentioned problems.

[0006] To achieve the above objectives, a comprehensive online monitoring method for power equipment based on multimodal artificial intelligence is provided in a first aspect of the present invention, the method comprising the following steps:

[0007] S1. Collect sample datasets of power equipment synchronously through the mobile inspection platform, bind the equipment, and generate a power equipment number to identify the equipment to which this sample belongs;

[0008] S2. Perform multimodal fusion recognition on the edge AI inference platform, combine the sample dataset, output the status anomaly category of the device to which the current sample belongs and the behavior anomaly category identified in the current image, and bind them with the power equipment number, spatial coordinates and timestamp to which the current sample belongs, to generate a standard event description quintuple;

[0009] S3. Based on the standard event description quintuple, a causal graph is constructed by combining the device electrical topology, spatial distance and event sequence. The abnormal state category of the device to which the current sample belongs and the abnormal behavior category identified in the current image are used as input. The causal score between nodes is calculated through structural connection enhancement and spatial distance attenuation mechanism to generate an abnormal causal graph with causal score and a causal path set for each abnormal state node.

[0010] S4. Map the causal graph and the causal path set of each abnormal node to a three-dimensional digital twin model to achieve anomaly space labeling, dynamic path backtracking, and interactive diagnosis.

[0011] Furthermore, S1 specifically includes:

[0012] The mobile inspection platform synchronously collects image frames, thermal images, status data, spatial coordinates, and timestamps of power equipment; the status data includes equipment surface temperature, SF6 gas concentration, and partial discharge intensity characteristic values.

[0013] The substation structure model is preloaded and bound to the equipment to generate a power equipment number to identify the device to which this sampling belongs;

[0014] After the device binding is completed, the image frame, thermal image, status data, spatial coordinates, timestamp, and the power equipment number used to identify the device to which this sampling belongs are encapsulated into a sample dataset.

[0015] Furthermore, the pre-loaded substation structure model is bound to the equipment to generate a power equipment number to identify the location of this sampling, specifically including:

[0016] The system preloads a substation structure model, in which each device is defined with a central spatial point and bound to a unique device number;

[0017] During its movement, the mobile inspection platform collects image frames, thermal images, status data, spatial coordinates, and timestamps every second, and obtains its current location. By combining the location of the central spatial point and calculating the Euclidean distance between the current location and the location of the central spatial point of the equipment, it binds the data to obtain the power equipment number to which this sampling belongs.

[0018] Furthermore, the edge AI inference platform employs a dual-path fusion neural network model for real-time identification; wherein, the dual-path fusion neural network model includes a fusion state identification module and a behavior detection module, the fusion state identification module is used to determine whether the device is in an abnormal operating condition; the behavior detection module is used to analyze whether there is any violation of regulations by personnel in the image.

[0019] Furthermore, the fusion state recognition module performs the following:

[0020] Visual feature vectors and features mapped from state data are extracted from the sample dataset.

[0021] The visual feature vector and the features mapped from the numerical state variables are concatenated to generate a multimodal fusion feature vector;

[0022] The multimodal fusion feature vector is input into the Softmax discriminant layer, and a device structure consistency regularization term is introduced for constraint to generate the state anomaly category of the device to which the current sample belongs;

[0023] The fusion status recognition module performs the following:

[0024] The image frame is input into a lightweight behavior classification network to generate the behavior anomaly category identified in the current image.

[0025] Furthermore, the construction of a causal graph based on the standard event description quintuple, combined with the device electrical topology, spatial distance, and event timing, specifically includes:

[0026] Generate a node set by using the standard event description quintuple as nodes;

[0027] Based on the nodes, causal candidate edges that meet the construction conditions are constructed. The construction conditions include: the occurrence time of the first node is less than that of the second node; the spatial distance between the first node and the second node does not exceed the preset maximum interaction distance; and the power equipment numbers corresponding to the first node and the second node, which are used to identify the power equipment to which this sampling belongs, are directly connected in the structural model by electrical, control or spatial means.

[0028] The fusion causal weights are calculated for causal candidate edges based on time interval features, spatial distance features, and the connection relationships between devices;

[0029] A causal graph is constructed based on the node set, the causal candidate edge set, and the fused causal weights.

[0030] Furthermore, based on the causal graph, upstream causal path backtracking is performed on each abnormal node with an abnormal state, all current nodes with fused causal weights higher than a preset threshold are filtered out, and the paths between the current node and the abnormal node are recorded to form a causal path set for each abnormal node in each state.

[0031] Furthermore, mapping the causal graph and the set of causal paths for each anomalous state node to a three-dimensional digital twin model specifically includes:

[0032] Bind the device model nodes in the digital twin 3D model of the causal graph;

[0033] Map node types to 3D annotations and bind them to time t for time playback control;

[0034] Obtain the causal path set for each abnormal node, connect them sequentially in three-dimensional space in chronological order to form a spatial path trajectory, and calculate the total importance score of the path by combining the fused causal weights.

[0035] The top K most important paths, ranked by their total importance score, are selected as the default highlighted paths.

[0036] Furthermore, the binding of device model nodes in the digital twin 3D model of the causal graph specifically includes:

[0037] Based on the power equipment number and corresponding spatial coordinates used to identify the location of this sampling, the model object is matched from the station structure model using a matching function, and a three-dimensional model entity object is generated.

[0038] A second aspect of the present invention provides a comprehensive online monitoring system for power equipment based on multimodal artificial intelligence, the system comprising:

[0039] The multimodal data acquisition unit is used to synchronously collect sample datasets of power equipment through the mobile inspection platform, bind the equipment, and generate a power equipment number to identify the equipment to which this sample belongs.

[0040] The edge deployment unit is used to perform multimodal fusion recognition on the edge AI inference platform. Combining the sample dataset, it outputs the status anomaly category of the device to which the current sample belongs and the behavior anomaly category identified in the current image. It is also bound to the power equipment number, spatial coordinates and timestamp to identify the power equipment to which this sample belongs to, and generates a standard event description quintuple.

[0041] The causal inference unit is used to construct a causal graph based on the standard event description quintuple, combined with the device electrical topology, spatial distance and event sequence. The current sample's state anomaly category and the behavior anomaly category identified in the current image are used as input. The causal score between nodes is calculated through structural connectivity enhancement and spatial distance attenuation mechanisms to generate an anomaly causal graph with causal scores and a set of causal paths for each state anomaly node.

[0042] The twin display unit is used to map the causal graph and the causal path set of each abnormal node to a three-dimensional digital twin model to realize anomaly space annotation, dynamic path backtracking and interactive diagnosis.

[0043] The beneficial technical effects of the present invention are at least as follows:

[0044] This invention proposes a comprehensive online monitoring method and system for power equipment based on multimodal artificial intelligence. It constructs a full-link monitoring solution encompassing synchronous on-site data acquisition, joint identification of status and behavior, temporal structure causal modeling, and spatial digital twin-driven display. This systematically solves the problems of data fragmentation, unstable identification, untraceable causality, and lack of visual interaction support in traditional systems. By structurally and synchronously binding image frames, thermal images, and equipment status variables during the data collection process using inspection robots or drones, spatial anchoring and temporal alignment at the equipment level are achieved, providing a unified data foundation for the identification model. The edge-side fusion identification model can jointly judge abnormal equipment status and human operational behavior in real time, outputting structured abnormal events bound with equipment numbers, timestamps, and spatial coordinates. The system further constructs a spatiotemporal graph based on the physical connection structure of the equipment and the temporal sequence of behavioral events, and introduces spatial attenuation and topology enhancement mechanisms to achieve causal relationship scoring and path backtracking for multimodal events. Based on the graph, the digital twin engine completes event spatial labeling through equipment number and coordinate matching, and achieves interactive tracing and auxiliary decision-making for operation and maintenance scenarios through trajectory rendering and behavior chain highlighting. The overall system features strong real-time performance, strong interpretability, and high deployment flexibility, making it suitable for complex substation structures, multi-device scenarios, and large-scale data access situations. It provides a complete solution for building a new generation of highly reliable, traceable, and visualized smart substation operation and maintenance system. Attached Figure Description

[0045] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0046] Figure 1 This is a flowchart of the integrated online monitoring method for power equipment based on multimodal artificial intelligence, as described in this invention. Detailed Implementation

[0047] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0048] like Figure 1 As shown in the embodiment of the present invention, a comprehensive online monitoring method for power equipment based on multimodal artificial intelligence is provided. The method includes:

[0049] S1. Collect sample datasets of power equipment synchronously through the mobile inspection platform, bind the equipment, and generate a power equipment number to identify the equipment to which this sample belongs.

[0050] Specifically, this step is used to collect multimodal data related to the operating status of primary equipment during the inspection of power substations, and to construct a structured sample set with spatial anchoring, time synchronization and equipment numbering.

[0051] This step relies on the following five types of information collected synchronously by the inspection robot or drone during the inspection process:

[0052] 1. Image Frame I t The images, acquired via an industrial high-definition visible light camera, are used to capture the appearance of the equipment. The resolution is 1920×1080, and the images are used for subsequent visual feature extraction.

[0053] 2. Thermal image H t Acquired by an infrared thermal imager and registered with image frames, it is used to detect hot spots and uneven heating phenomena in equipment;

[0054] 3. Status data:

[0055] θ t Surface temperature of the equipment is obtained from the thermal resistors installed on key components;

[0056] C t SF6 gas concentration, obtained from a sensor embedded in the GIS cavity;

[0057] U t Partial discharge intensity characteristic value, taken from UHF partial discharge sensor;

[0058] 4. Spatial coordinates T t The coordinates of the patrol platform within the station are calculated by the SLAM module in combination with vision and wheeled odometer.

[0059] 5. Timestamp t: Generated by a unified synchronization system to ensure that all modal data are aligned based on a unified time axis.

[0060] Furthermore, the system preloads a substation structural model, in which each primary device (such as GIS branch, circuit breaker, transformer connection side) is defined with a central spatial point P. j The platform is assigned a unique device number. During its movement, the platform collects images, thermal images, and status data every second, and obtains its current location (T). t Device pairing is performed in the following ways:

[0061]

[0062] Among them, Device_ID t : The device number corresponding to the current sampling point; T t The current spatial coordinates of the inspection platform are output by the SLAM positioning system; P j : The center spatial coordinates of the j-th device in the structural model; ||T t -P j ||: Euclidean distance between the current coordinates and the device center point; argmin j : This indicates that the device number with the smallest distance is being searched among all devices;

[0063] Furthermore, after completing device binding, all current sensing data will be uniformly encapsulated into structured sample D. t The format is as follows:

[0064] D t =(Device_ID) t ,T t ,t,I t H t ,θ t C t U t (2)

[0065] Among them, D t : Represents a set of structured data samples collected at the current time t; Device_ID t : The device number to which the current data belongs; T t : Current 3D spatial coordinates; t: Timestamp; I t Visible light image frame; H t Thermographic image; θ t Temperature value; C t : Gas concentration; U t Partial discharge intensity;

[0066] Furthermore, all samples D t The dataset D = {D} tThe samples are organized into a time-series sample stream in the edge host cache for direct reading by the model module in the next step.

[0067] For example: When the robot is located in the area west of the substation, its current position is T. t = (8.45, 2.30, 1.65), which has the smallest distance from location point P8 = (8.47, 2.29, 1.64) in the "GIS Circuit Breaker A1" model, the system determines the current device number as "GIS_A1" and packages all sampled data into sample D. t The images show a blackened surface, localized overheating in the thermal image, and significantly elevated partial discharge signals. All of these are attributed to this device and written into the dataset.

[0068] Final output: Device_ID t : Used to identify the power equipment number to which this sampling belongs; D: Structured sample dataset, containing images, thermal images, state values, spatial and temporal information required by all subsequent AI modules.

[0069] S2. Perform multimodal fusion recognition on the edge AI inference platform. Combine the sample dataset to output the status anomaly category of the device to which the current sample belongs and the behavior anomaly category identified in the current image. Bind these categories with the power equipment number used to identify the power equipment to which this sample belongs, spatial coordinates, and timestamp to generate a standard event description quintuple.

[0070] Specifically, this step converts the multimodal information in the structured sample D collected in step one into equipment status anomaly labels L. t And abnormal behavior label A t This step involves more than just simple image recognition or sensor threshold judgment; it utilizes edge AI technology for fusion recognition based on the multimodal synchronous acquisition structure proposed in this invention. Considering that power equipment anomalies are characterized by "weak signals, high interference, and heterogeneous distribution," while human behavior anomalies have attributes such as "fuzzy boundaries, semantic diversity, and scene dependence," traditional classification models struggle to simultaneously address these characteristics. Therefore, this step introduces a multimodal heterogeneous feature matching mechanism and a structured device binding regularization term into the model design, constructing an edge recognition method for complex operating scenarios in power substations. By identifying whether each device has potential risks in its state or behavior at the current time point, this step outputs structured preliminary event results, providing event nodes for subsequent causal graph inference. This is the key entry point for the "data-driven intelligent diagnostic chain" in this invention.

[0071] Furthermore, the system is deployed on an edge AI inference platform (such as Atlas300) and uses a dual-path fusion neural network model for real-time recognition. The overall process includes two modules: one is a fusion status recognition module, used to determine whether the device is in an abnormal operating condition; the other is a behavior detection module, used to analyze whether there is any illegal behavior by people in the image.

[0072] (1) Status Anomaly Identification Module

[0073] The input is state data [θ] t C t U t and visual feature vector v img The latter comes from the image frame I t With thermal image H t CNN feature extraction. This invention uses a three-layer convolutional structure to extract low-level texture features such as hotspot regions and contour sharpness. The convolutional kernel size is 3×3, and the number of channels is 16, 32, and 64. The output features are compressed to v. img (Dimension 64)

[0074] The state data is mapped to a feature vector v through a two-layer MLP extractor. stat (Dimensionality is 32). The two are concatenated at the fusion layer to form a unified representation:

[0075] v t =ReLU(W f [v img ∥v stat ]+b f (3)

[0076] Among them, v img : Visual feature vectors extracted from images and thermal images; v stat : Features mapped from numerical state variables; ∥: Vector concatenation; W f ,b f Weights and biases of the fully connected fusion layer; v t : Final multimodal fusion feature vector.

[0077] The fused feature is fed into the Softmax discriminant layer to output an anomaly classification label L. t This includes categories such as "normal," "abnormal temperature rise," "SF6 leakage," and "abnormal partial discharge." Simultaneously, to adapt to the actual scenarios of power equipment monitoring, a device structure consistency regularization term Ω is introduced. struct This is used to enhance the model's ability to learn how to cluster similar abnormal behaviors among similar devices:

[0078]

[0079] in, Anomaly prediction labels output by the model; Ω struct λ1: Structural consistency regularization term, constraining the feature distribution distance between devices of the same type; λ1: Regularization weight, set to 0.05. CrossEntropy is the cross-entropy loss function.

[0080] Furthermore, this regularization term is achieved by recording the mean μ of the characteristic distribution of similar devices under normal conditions. k For all samples of the same type, v t Strengthen the convergence and improve cross-device model consistency to solve the problem of unstable recognition caused by sample bias.

[0081] (2) Behavioral Anomaly Detection Module

[0082] Behavior recognition is mainly based on image frame I t Considering the computational resource limitations of edge deployment, a lightweight convolutional detector (such as YOLOv5-Nano) is used to identify human targets in images, and the detected bounding boxes are then fed into an action classifier.

[0083] This invention does not use complex temporal models, but directly segments the target region and feeds it into a lightweight behavior classification network (two-layer CNN + fully connected layer) to determine whether the region contains the following behavior label A. t :

[0084] Not wearing a safety helmet; hands crossing the warning line; non-inspection personnel entering the high-voltage area; bringing tools into the control cabinet area without proper equipment.

[0085] The detection region is further enhanced by aligning with the spatial grid built into the image. Each behavior label result A t Bound to timestamp t and spatial coordinates T t , with status label L t Together they constitute the results of a multimodal event.

[0086] Finally, output L. t : The current sample belongs to the abnormal status category of the device; A t : The category of behavioral anomalies identified in the current image; Output events are uniformly bound to Device_ID t T t Together with t, they form the standard event description quintuple (L... t A t Device_ID t ,T t ,t).

[0087] S3. Based on the standard event description quintuple, a causal graph is constructed by combining the device electrical topology, spatial distance and event sequence. The abnormal state category of the device to which the current sample belongs and the abnormal behavior category identified in the current image are used as inputs. The causal score between nodes is calculated through structural connection enhancement and spatial distance attenuation mechanism to generate an abnormal causal graph with causal score and a causal path set for each abnormal state node.

[0088] Specifically, this step uses the multimodal recognition results output from step two as input, combines the structural characteristics of power equipment with the temporal dependence of operation and maintenance behaviors, constructs an anomaly causal graph, and performs path reasoning to clarify the potential causal relationship between equipment state anomalies and behavioral events. Considering the special characteristics of power systems—such as the defined electrical and spatial topology between primary equipment, the propagation of equipment anomalies, and the typical time delay effect of abnormal behaviors—this step proposes an innovative graph structure construction method that combines equipment topology mapping and behavioral temporal causal graph construction. Furthermore, it introduces physical constraint regularization terms and local chain decay mechanisms into graph representation learning to construct realistic and reliable causal reasoning paths. This not only serves for display and backtracking but also provides a foundation for subsequent alarm priority assessment and auxiliary operation and maintenance decisions.

[0089] Furthermore, within a fixed time window (e.g., 30 seconds), all occurring state anomalies and behavioral events are organized into an event node set V. t Each node v i ∈V t Includes quintuple properties (L t A t Device_ID t ,T t ,t). Edge E between nodes t It is a potential causal path, and the key to its construction lies in combining the following three types of conditions:

[0090] Time condition: node v i The occurrence time is less than node v j , i.e. t i <t j ;

[0091] Spatial conditions: node v i With v j Spatial distance || T i -T j ∥Do not exceed the preset maximum interaction distance;

[0092] Structural conditions: node v i With v j Device ID i With Device_ID jThe structural model contains direct electrical, control, or spatial connections;

[0093] After establishing causal candidate edges, this invention calculates a fused causal weight α for each edge. ij The definition is as follows:

[0094] α ij =σ(w T [h i ∥h j ∥Δt ij ∥Δd ij ]+γ·I conn (i,j))-λ·∥T i -T j ∥ 2 (5)

[0095] Where, α ij Event node v i For v j The causal contribution of h is a factor; a larger value indicates a stronger causal relationship. i ,h j : These are the embedding representations of nodes i and j, derived from vectors (32-dimensional) encoded with their event type and device number, generated through a two-layer MLP; Δt ij =t j -t i : Time interval characteristics, compressed through log transform; Δd ij =∥T i -T j ∥: Spatial distance characteristics; I conn (i,j): Structural connectivity labeling function, 1 if the two devices are connected in the topology graph, 0 otherwise; γ: Structural connectivity enhancement factor, used to improve the causal score between physically directly connected devices (such as between the main bus and branches in GIS); λ: Spatial attenuation coefficient, used to weaken the causal score between distant events; σ: Sigmoid activation function; [·∥·]: Represents vector concatenation operation; w: Learnable weight vector (dimension matching concatenation result).

[0096] Furthermore, this formula adds two unique physical scene terms to the traditional graph attention model:

[0097] Structural enhancement term γ·I conn (i,j): Clearly utilize the characteristics of "physical connection → signal linkage" between power equipment to improve the score of certain edges;

[0098] Spatial attenuation term -λ·∥T i -T j ∥ 2: To prevent events that are geographically distant but temporally close from being mistakenly identified as causal paths;

[0099] For example, in practical applications, if an operator approaches the "GIS_A1" device without wearing a safety helmet, and the system detects a sudden change in SF6 concentration in the device one second later, then this behavioral event node and the abnormal state node have a triple relationship of "time sequence, spatial proximity, and equipment consistency," α ij A score of 0.85 or higher is recorded as a causal reasoning path.

[0100] Subsequently, based on the graph structure G t =(V t E t ,α), the system for each abnormal state node v j Perform upstream causal path backtracking and filter all α ij v above a threshold (e.g., 0.6) i Nodes are recorded as reasoning paths:

[0101] P j ={v i |α ij >θ causal} (6)

[0102] Among them, P j Node v j The set of causal upstream events; θ causal Causality scoring threshold, set at 0.6–0.8; v i These are usually upstream abnormal behavior events, such as unauthorized approach or accidental entry into a high-voltage area.

[0103] It is important to note that the most innovative aspect of this step lies in starting from the engineering constraints of the power industry and explicitly encoding three physical logical factors—"time sequence," "spatial distance," and "equipment structural connection"—into a graph attention mechanism. This achieves an interpretable causal modeling method for substation equipment anomaly analysis scenarios. The designed causal scoring formula incorporates two mechanisms unique to this invention's scenario within the variable control range: a structural enhancement term and a spatial attenuation term. This makes the system more targeted in identifying causal paths, reduces false connections, and possesses good scalability and operational controllability in practical engineering.

[0104] S4. Map the causal graph and the causal path set of each abnormal node to a three-dimensional digital twin model to achieve anomaly space labeling, dynamic path backtracking, and interactive diagnosis.

[0105] Specifically, this step aims to transform the abnormal causal graph G output in step three... t and the set of causal paths P for each node in an abnormal state jMapped to a 3D digital twin platform, it realizes a visualized and interactive anomaly diagnosis and display system. Its task is to accurately map the completed identification results and causal relationships to the specific device location in 3D space based on attributes such as space, time, equipment structure, and anomaly type. It also provides human-computer interaction interfaces, including anomaly hotspot highlighting, causal path backtracking playback, and anomaly behavior pop-up prompts, to help on-site operation and maintenance personnel quickly understand the background and evolution path of the anomaly.

[0106] Furthermore, the first step in this invention is to process each graph node v i ∈G t The system binds the device model node to the digital twin 3D model. This is based on the device ID (Device_ID). t and spatial coordinates T t Match model objects from the site's structural model (imported from a CAD or BIM platform). Implement the following space binding:

[0107]

[0108] in, This represents the k-th 3D model entity object within the site; Device_ID t : The device number to which the node belongs; T t : The three-dimensional spatial coordinates of the node within the station; match(·): represents the matching function, which prioritizes matching by device number. If duplicate numbers exist, it further compares the spatial coordinate errors and takes the minimum value.

[0109] Furthermore, after binding is complete, the system can map node types to 3D labels (e.g., abnormal status is labeled with a red tag, and abnormal behavior is labeled with an orange icon), and bind time t for time playback control.

[0110] The second step is to construct the abnormal path trajectory. The system reads P. j Each abnormal node v in the middle j The corresponding upstream path Connect these elements sequentially in 3D space to form a spatial path. The path is rendered in space as a directional curve or arrow, and a time-controlled slider is provided for progressive playback. For example, if v i1 The incident was related to "personnel approaching GIS_B3". If the error is "GIS_B3SF6 leak", then the T values ​​of both are... t The points are (x1, y1, z1) and (x2, y2, z2), respectively. The path segment is rendered as a curve connecting these two points in the 3D platform and flashes and highlights as time progresses during playback.

[0111] To ensure the display focuses on the most critical anomaly events, the system is based on each path P. j The causal strength in the path is used to calculate the total importance score S. j The formula is as follows:

[0112]

[0113] Among them, S j : Node v with abnormal state j The path display weight; α ij : Causal edge score from the graph; ω(L j ): Weighting function for abnormal status types, such as assigning a value of 1.2 to "partial discharge abnormality", 1.0 to "temperature rise abnormality", and 1.5 to "SF6 leakage";

[0114] The system uses S j The top k most important paths are selected for default highlighting, while the remaining paths are hidden by default and only expanded when the user interacts with them.

[0115] The visualization platform provides the following basic interactive functions:

[0116] Clicking on the device model will bring up details of the current status exception, including exception type L. t Occurrence time t, corresponding behavior node;

[0117] Select the behavior node to trace back all downstream state nodes affected by it and display the scope of the impact;

[0118] Drag the timeline to view the abnormal evolution within the past Δt time window;

[0119] Adjust the "Path Weight Threshold" slider to customize the number and range of paths displayed.

[0120] The system also supports exporting the causal propagation process of a certain abnormal event in video format, providing materials for subsequent operation and maintenance records and analysis.

[0121] For example: The system detected a sudden increase in SF6 gas concentration (L) at t=2025.07.1514:32:45 for device "GIS_A4". t The source tracing path is that someone approached the high-voltage area (A) 2 minutes ago. t The platform will highlight the behavior node in red on the GIS_A4 model and draw an arc from the behavior node location to the device. During playback, the path will be highlighted and flashed, while event tags and scoring information (such as "α") will be displayed on the side. ij =0.87,S j =1.31”).

[0122] This invention also provides a comprehensive online monitoring system for power equipment based on multimodal artificial intelligence, the system comprising:

[0123] The multimodal data acquisition unit is used to synchronously collect sample datasets of power equipment through the mobile inspection platform, bind the equipment, and generate a power equipment number to identify the equipment to which this sample belongs.

[0124] The edge deployment unit is used to perform multimodal fusion recognition on the edge AI inference platform. Combining the sample dataset, it outputs the status anomaly category of the device to which the current sample belongs and the behavior anomaly category identified in the current image. It is also bound to the power equipment number, spatial coordinates and timestamp to identify the power equipment to which this sample belongs to, and generates a standard event description quintuple.

[0125] The causal inference unit is used to construct a causal graph based on the standard event description quintuple, combined with the device electrical topology, spatial distance and event sequence. The current sample's state anomaly category and the behavior anomaly category identified in the current image are used as input. The causal score between nodes is calculated through structural connectivity enhancement and spatial distance attenuation mechanisms to generate an anomaly causal graph with causal scores and a set of causal paths for each state anomaly node.

[0126] The twin display unit is used to map the causal graph and the causal path set of each abnormal node to a three-dimensional digital twin model to realize anomaly space annotation, dynamic path backtracking and interactive diagnosis.

[0127] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0128] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or units may be electrical, mechanical, or other forms.

[0129] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0130] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A comprehensive online monitoring method for power equipment based on multimodal artificial intelligence, characterized in that, The method includes the following steps: S1. Collect sample datasets of power equipment synchronously through the mobile inspection platform, bind the equipment, and generate a power equipment number to identify the equipment to which this sample belongs; S2. Perform multimodal fusion recognition on the edge AI inference platform, combine the sample dataset, output the status anomaly category of the device to which the current sample belongs and the behavior anomaly category identified in the current image, and generate a standard event description quintuple with the power equipment number, spatial coordinates and timestamp used to identify the power equipment to which this sample belongs; S3. Based on the standard event description quintuple, a causal graph is constructed by combining the device electrical topology, spatial distance and event sequence. The abnormal state category of the device to which the current sample belongs and the abnormal behavior category identified in the current image are used as input. The causal score between nodes is calculated through structural connection enhancement and spatial distance attenuation mechanism to generate an abnormal causal graph with causal score and a causal path set for each abnormal state node. S4. Map the causal graph and the causal path set of each abnormal state node to a three-dimensional digital twin model to achieve anomaly space labeling, dynamic path backtracking and interactive diagnosis. Specifically, the construction of a causal graph based on the standard event description quintuple, combined with the device's electrical topology, spatial distance, and event sequence, includes: Generate a node set by using the standard event description quintuple as nodes; Based on the nodes, causal candidate edges that meet the construction conditions are constructed. The construction conditions include: the occurrence time of the first node is less than that of the second node; the spatial distance between the first node and the second node does not exceed the preset maximum interaction distance; and the power equipment numbers corresponding to the first node and the second node, which are used to identify the power equipment to which this sampling belongs, are directly connected in the structural model by electrical, control or spatial means. The fusion causal weights are calculated for causal candidate edges based on time interval features, spatial distance features, and the connection relationships between devices; A causal graph is constructed based on the node set, the causal candidate edge set, and the fused causal weights. The process of mapping the causal graph and the set of causal paths for each anomalous node to a three-dimensional digital twin model specifically includes: The nodes of the causal graph are bound to the device model nodes in the digital twin 3D model; Map node types to 3D annotations and bind them to time t for time playback control; Obtain the causal path set for each abnormal node, connect them sequentially in three-dimensional space in chronological order to form a spatial path trajectory, and calculate the total importance score of the path by combining the fused causal weights. The top K most important paths, ranked by their total importance score, are selected as the default highlighted paths.

2. The method for comprehensive online monitoring of power equipment based on multimodal artificial intelligence according to claim 1, characterized in that, S1 specifically includes: The mobile inspection platform synchronously collects image frames, thermal images, status data, spatial coordinates, and timestamps of power equipment; among which, the status data includes equipment surface temperature, SF6 gas concentration, and partial discharge intensity characteristic values. The substation structure model is preloaded and bound to the equipment to generate a power equipment number to identify the device to which this sampling belongs; After the device binding is completed, the image frame, thermal image, status data, spatial coordinates, timestamp, and the power equipment number used to identify the device to which this sampling belongs are encapsulated into a sample dataset.

3. The method for comprehensive online monitoring of power equipment based on multimodal artificial intelligence according to claim 2, characterized in that, The pre-loaded substation structure model is bound to the equipment to generate a power equipment number that identifies the equipment to which this sampling belongs, specifically including: The system preloads a substation structure model, in which each device is defined with a central spatial point and bound to a unique device number; During its movement, the mobile inspection platform collects image frames, thermal images, status data, spatial coordinates, and timestamps every second, and obtains its current location. By combining the location of the central spatial point and calculating the Euclidean distance between the current location and the location of the central spatial point of the equipment, it binds the data to obtain the power equipment number to which this sampling belongs.

4. The method for comprehensive online monitoring of power equipment based on multimodal artificial intelligence according to claim 1, characterized in that, The edge AI inference platform uses a dual-path fusion neural network model for real-time identification; wherein, the dual-path fusion neural network model includes a fusion state identification module and a behavior detection module, the fusion state identification module is used to determine whether the device is in an abnormal operating condition; the behavior detection module is used to analyze whether there is any illegal behavior by personnel in the image.

5. The integrated online monitoring method for power equipment based on multimodal artificial intelligence according to claim 4, characterized in that, The fusion status recognition module performs the following: Visual feature vectors and features mapped from state data are extracted from the sample dataset. The visual feature vector and the features mapped from the numerical state variables are concatenated to generate a multimodal fusion feature vector; The multimodal fusion feature vector is input into the Softmax discriminant layer, and a device structure consistency regularization term is introduced for constraint to generate the state anomaly category of the device to which the current sample belongs; The fusion status recognition module performs the following: Image frames are fed into a lightweight behavior classification network to generate the categories of behavioral anomalies identified in the current image.

6. The method for comprehensive online monitoring of power equipment based on multimodal artificial intelligence according to claim 1, characterized in that, Based on the causal graph, upstream causal path backtracking is performed on each abnormal node with an abnormal state. All current nodes with fused causal weights higher than a preset threshold are filtered out, and the paths between the current node and the abnormal node are recorded to form a causal path set for each abnormal node in each state.

7. The method for comprehensive online monitoring of power equipment based on multimodal artificial intelligence according to claim 1, characterized in that, The binding of nodes in the causal graph to device model nodes in the digital twin 3D model specifically includes: Based on the power equipment number and corresponding spatial coordinates used to identify the location of this sampling, the model object is matched from the station structure model using a matching function, and a three-dimensional model entity object is generated.

8. A system for implementing the integrated online monitoring method for power equipment based on multimodal artificial intelligence as described in claim 1, characterized in that, The system includes: The multimodal data acquisition unit is used to synchronously collect sample datasets of power equipment through the mobile inspection platform, bind the equipment, and generate a power equipment number to identify the equipment to which this sample belongs. The edge deployment unit is used to perform multimodal fusion recognition on the edge AI inference platform. Combining the sample dataset, it outputs the status anomaly category of the device to which the current sample belongs and the behavior anomaly category identified in the current image. It also generates a standard event description quintuple with the power equipment number, spatial coordinates and timestamp used to identify the power equipment to which this sample belongs. The causal inference unit is used to construct a causal graph based on the standard event description quintuple, combined with the device electrical topology, spatial distance and event sequence. The current sample's state anomaly category and the behavior anomaly category identified in the current image are used as input. The causal score between nodes is calculated through structural connectivity enhancement and spatial distance attenuation mechanisms to generate an anomaly causal graph with causal scores and a set of causal paths for each state anomaly node. The twin display unit is used to map the causal graph and the causal path set of each abnormal node to a three-dimensional digital twin model to realize anomaly space annotation, dynamic path backtracking and interactive diagnosis.

Images