A multi-scale multi-modal data fusion method for a power scenario and related devices

Abstract

This invention discloses a multi-scale, multi-modal data fusion method and related apparatus for power scenarios, relating to the field of intelligent operation and maintenance technology for power systems. Based on a spatiotemporal structure pyramid network as its core support, this invention significantly improves the feature utilization rate of multi-source heterogeneous data through hierarchical extraction and cross-fusion of multi-scale spatiotemporal features, strengthens the semantic association between cross-modal data, and avoids redundant collection of invalid data through targeted supplementary collection, further ensuring the integrity of key information. The resulting anomaly detection model significantly improves the diagnostic accuracy and positioning precision of typical defects in power equipment, and effectively enhances its robustness to complex field conditions such as noise interference and partial data loss, providing a reliable guarantee for the safe and stable operation and maintenance of power equipment.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance technology for power systems, and in particular to a multi-scale, multi-modal data fusion method and related apparatus for power scenarios. Background Technology

[0002] With the deepening of smart grid construction, the scale of power systems is expanding and the structure is becoming increasingly complex. Ensuring the safe, stable and reliable operation of power systems has become a major challenge for the industry. As a core means of discovering potential faults and preventing major accidents, the level of intelligence of power equipment inspection directly determines the efficiency and safety of power grid operation and maintenance.

[0003] In recent years, inspection technology has rapidly evolved from traditional manual methods to automation and intelligence. Various power sensing terminals and monitoring devices have been widely deployed, and the multi-dimensional sensing capabilities of power equipment have been greatly improved, generating massive amounts of heterogeneous multi-source data, providing a rich data foundation for equipment condition assessment and anomaly diagnosis.

[0004] Although existing technologies have unprecedented multi-dimensional perception capabilities, there are still significant shortcomings in the integration and utilization of multi-source data. It is difficult to effectively explore the inherent correlation between data from different sources and to fully unleash the collaborative value of multi-source perception data. As a result, the accuracy and robustness of subsequent equipment status assessment and anomaly diagnosis cannot meet the actual operation and maintenance needs of complex power scenarios, which has become a key bottleneck restricting the further development of intelligent power operation and maintenance technology. Summary of the Invention

[0005] This invention provides a multi-scale, multi-modal data fusion method and related apparatus for power scenarios, which solves the technical problems of difficulty in fusing multi-source heterogeneous data from existing power equipment and insufficient accuracy in anomaly diagnosis.

[0006] The first aspect of this invention provides a multi-scale, multi-modal data fusion method for power scenarios, comprising:

[0007] Collect multimodal data within the target power equipment area and preprocess it to obtain key multimodal data;

[0008] Anomaly diagnosis is performed on the key multimodal data using a pre-set spatiotemporal structure pyramid network;

[0009] Based on the localization results in the anomaly diagnosis results, target data is extracted from the key multimodal data, and data acquisition instructions that match the anomaly type in the anomaly diagnosis results are determined.

[0010] The data acquisition device, which is matched with the data acquisition command, is controlled to acquire data in the area corresponding to the positioning result, thereby obtaining valid detection data.

[0011] Map the target data and the effective detection data to a unified latent space;

[0012] A pre-defined graph neural network is used to fuse and model the data in the unified latent space to obtain an anomaly detection fusion model. The anomaly detection fusion model is then used to perform anomaly diagnosis and location of power equipment, generating diagnosis and location results.

[0013] Optionally, the spatiotemporal structure pyramid network includes a spatial pyramid sub-network, a temporal pyramid sub-network, and a spatiotemporal fusion processing sub-network. The step of using a pre-set spatiotemporal structure pyramid network to perform anomaly diagnosis on the key multimodal data includes:

[0014] The spatial pyramid subnetwork is used to extract features from the key multimodal data to obtain multi-scale spatial fusion features;

[0015] The key multimodal data are used to extract features using the temporal pyramid subnetwork to obtain temporal dynamic fusion features;

[0016] The spatiotemporal fusion processing subnetwork is used to perform spatiotemporal feature fusion and anomaly diagnosis on the multi-scale spatial fusion features and the temporal dynamic fusion features to obtain the anomaly diagnosis results.

[0017] Optionally, the spatial pyramid sub-network includes a convolutional neural network backbone, multi-level pooling units, and upsampling and concatenation layers. The multi-level pooling units include multiple cascaded pooling layers. The step of using the spatial pyramid sub-network to extract features from the key multimodal data to obtain multi-scale spatial fusion features includes:

[0018] The visible light image data and infrared image data in the key multimodal data are extracted using the backbone of the convolutional neural network to obtain an initial feature map;

[0019] The initial feature map is subjected to multi-scale pooling processing using the multiple cascaded pooling layers to obtain multiple sub-feature maps;

[0020] The upsampling and stitching layer is used to upsample multiple sub-feature maps according to a preset size, and the upsampling results are stitched together to obtain multi-scale spatial fusion features.

[0021] Optionally, the temporal pyramid sub-network includes a numerical feature extraction unit, a multi-timescale temporal convolution unit, and a multi-timescale feature fusion unit. The step of using the temporal pyramid sub-network to extract features from the key multimodal data to obtain temporally dynamic fused features includes:

[0022] The numerical feature extraction unit is used to extract numerical features from the visible light image data, infrared image data, and local temperature distribution map in the key multimodal data to obtain numerical features;

[0023] The voltage data, current data and numerical features in the key multimodal data are processed using the multi-timescale temporal convolutional unit to obtain dynamic features corresponding to different time scales.

[0024] The dynamic features are fused using the multi-timescale feature fusion unit to obtain time-dynamic fused features.

[0025] Optionally, mapping the target data and the valid detection data to a unified latent space includes:

[0026] The target data and the effective detection data are encoded to obtain multiple initial hidden vectors;

[0027] Dimensionality reduction is performed on all the initial latent vectors to obtain intermediate latent vectors;

[0028] All the intermediate latent vectors are mapped to the same feature space to construct a unified latent space.

[0029] Optionally, the step of using a pre-set graph neural network to fuse and model the data in the unified latent space to obtain an anomaly detection fusion model, and then using the anomaly detection fusion model to perform power equipment anomaly diagnosis and location, generating diagnosis and location results, includes:

[0030] Construct an intermodal relationship graph corresponding to each modality data in the unified latent space; the nodes of the intermodal relationship graph are each modality, and the node features are the intermediate latent vectors of the corresponding modality in the unified latent space;

[0031] Calculate the cosine similarity between nodes in the intermodal relationship graph and perform exponential normalization, then use the exponential normalization result as the edge weight.

[0032] Using a pre-built graph neural network, based on the inter-modal relationship graph and the edge weights, graph convolution fusion processing is performed on the intermediate latent vectors in the unified latent space to obtain the anomaly detection fusion model;

[0033] The anomaly detection fusion model is used to perform anomaly diagnosis and location of power equipment, and generate diagnosis and location results.

[0034] The second aspect of this invention provides a multi-scale, multi-modal data fusion system for power scenarios, comprising:

[0035] The acquisition module is used to acquire multimodal data within the target power equipment area and perform preprocessing to obtain key multimodal data;

[0036] The spatiotemporal structure pyramid network module is used to perform anomaly diagnosis on the key multimodal data using a pre-set spatiotemporal structure pyramid network.

[0037] The data extraction and processing module is used to extract target data from the key multimodal data based on the location results in the anomaly diagnosis results, and to determine the data acquisition instructions that match the anomaly type in the anomaly diagnosis results.

[0038] The data acquisition device module is used to control a data acquisition device that matches the data acquisition command to acquire data in the area corresponding to the positioning result, so as to obtain valid detection data.

[0039] The encoder network module is used to map the target data and the valid detection data to a unified latent space;

[0040] The graph neural network module is used to fuse and model the data in the unified latent space using a pre-set graph neural network to obtain an anomaly detection fusion model, and to perform power equipment anomaly diagnosis and location through the anomaly detection fusion model to generate diagnosis and location results.

[0041] A third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the multi-scale multimodal data fusion method for power scenarios as described above.

[0042] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the multi-scale, multi-modal data fusion method for power scenarios as described above.

[0043] The fifth aspect of the present invention provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer performs the multi-scale multimodal data fusion method for power scenarios as described above.

[0044] As can be seen from the above technical solutions, the present invention has the following advantages:

[0045] This invention provides a multi-scale, multi-modal data fusion method and related apparatus for power scenarios. First, multi-modal sensing data of the target power equipment area is preprocessed to form regular key multi-modal data. Then, a spatiotemporal pyramid network is used to extract multi-scale features across spatiotemporal dimensions and perform preliminary anomaly diagnosis on the key multi-modal data. This overcomes the limitations of traditional single-scale feature processing, effectively mining the operational status information of multi-modal data at different spatial and temporal scales, and constructing a highly correlated spatiotemporal feature system. This solves the problem of fusion difficulties caused by the fragmentation of spatiotemporal features and insufficient semantic correlation of multi-source heterogeneous data, while providing accurate diagnostic guidance and a high-quality feature foundation for subsequent data processing. Based on this, combined with the preliminary diagnostic results of the spatiotemporal structure pyramid network, corresponding target data is extracted and appropriate data acquisition instructions are determined. Related acquisition equipment is then controlled to conduct supplementary detection of the abnormal location area to obtain effective detection data. The target data and effective detection data are then mapped to a unified latent space, achieving semantic alignment and feature unification of heterogeneous data. This mapping process forms a highly efficient synergy with the spatiotemporal structure pyramid network. The accurate features and diagnostic basis provided by the spatiotemporal structure pyramid network ensure that the data mapped to the unified latent space has high correlation and high effectiveness. Furthermore, the mapping of the unified latent space further breaks down the heterogeneous barriers of multimodal data, allowing the features extracted by the spatiotemporal structure pyramid network to fully leverage their synergistic value, completely solving the pain point of difficulty in fusing multi-source heterogeneous data in existing technologies. The organic combination of these two methods significantly improves the comprehensive utilization efficiency of multimodal data, strengthens the anomaly detection model's ability to represent the state of power equipment, effectively compensates for the insufficient diagnostic accuracy of traditional methods, significantly improves the accuracy and reliability of power equipment anomaly diagnosis, and enhances the adaptability of the detection model under complex operating conditions, providing a solid technical guarantee for the stable operation and intelligent maintenance of power equipment. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the steps of a multi-scale, multi-modal data fusion method for power scenarios provided in Embodiment 1 of the present invention.

[0048] Figure 2 This is a schematic diagram of the spatial pyramid sub-network provided in Embodiment 2 of the present invention;

[0049] Figure 3This is a schematic diagram of the time pyramid subnetwork provided in Embodiment 2 of the present invention;

[0050] Figure 4 This is a structural block diagram of a multi-scale, multi-modal data fusion system for power scenarios provided in Embodiment 3 of the present invention;

[0051] Figure 5 This is a structural block diagram of an electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0052] This invention provides a multi-scale, multi-modal data fusion method and related apparatus for power scenarios, which solves the technical problems of difficulty in fusing multi-source heterogeneous data from existing power equipment and insufficient accuracy in anomaly diagnosis.

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] This invention addresses the challenges of existing technologies, such as difficulties in data-level fusion, significant heterogeneity barriers, simplistic feature fusion strategies, lack of cross-modal semantic association, and insufficient utilization of spatiotemporal multi-scale characteristics. It proposes a multi-scale, multi-modal data fusion scheme for power scenarios. This technical solution implements a cognitive logic of "macroscopic investigation → microscopic focus → comprehensive modeling," achieving efficient and accurate data fusion and analysis. The anomaly detection fusion model obtained through data fusion processing achieves a fault detection consistency rate of ≥92% and reduces response time by 60%. Specifically, the spatiotemporal structure pyramid network analyzes the spatial and temporal (trend, sequence) characteristics of data at multiple scales and levels, facilitating the subsequent construction of anomaly detection fusion models capable of accurately diagnosing power equipment anomalies. This model effectively detects and diagnoses various anomalies, from local to global and from instantaneous to continuous. The encoder network creates a low-dimensional latent space specifically optimized for anomaly patterns, where different types of anomalies form well-separable clusters. Graph neural networks, a graph convolutional fusion method based on inter-modal relationship graphs, effectively utilize the complex relationships between multimodal data, improving the accuracy and robustness of anomaly detection.

[0055] Example 1:

[0056] Please see Figure 1 , Figure 1The flowchart illustrates the steps of a multi-scale, multi-modal data fusion method for power scenarios provided in Embodiment 1 of the present invention.

[0057] This invention provides a multi-scale, multi-modal data fusion method for power scenarios, comprising:

[0058] Step 101: Collect multimodal data within the target power equipment area and preprocess it to obtain key multimodal data.

[0059] In this embodiment of the invention, firstly, an inspection robot is used to collect multimodal data within the target power equipment area. The multimodal data includes visible light image data, infrared image data, satellite remote sensing surface temperature data, voltage data, and current data collected over a period of time. After the data collection is completed, the lightweight computing node integrated into the inspection robot is used to preprocess the multimodal data to obtain key multimodal data.

[0060] The specific preprocessing steps include:

[0061] (1) For visible light image data, the lightweight computing node identifies key equipment areas through edge detection algorithm, extracts visible light image data of key equipment areas, removes irrelevant background information, and focuses on the representation of the equipment body status;

[0062] (2) For infrared image data, the lightweight computing node identifies the overheated area of ​​the device through threshold and clustering algorithm, and extracts infrared image data with abnormal temperature. At the same time, the extracted visible light image data and infrared image data are spatially aligned to ensure the consistency of image data in spatial dimension.

[0063] (3) For satellite remote sensing surface temperature data, the lightweight computing node extracts the surface temperature value from the data and converts it into a standard data format. Then, it matches the temperature data with the location coordinates of the inspection robot and generates a local temperature distribution map based on the matching results, thereby realizing the spatialization and structuring of surface temperature data.

[0064] (4) For voltage and current data, the lightweight computing node uses a low-pass filter to remove high-frequency noise and transient interference from the data, then identifies and removes abnormal readings based on statistical methods, and finally derives key electrical features based on the data after the abnormality removal. Key electrical features include power, power factor and harmonic content, thereby realizing the feature extraction of electrical data.

[0065] After the above preprocessing, the key multimodal data obtained include preprocessed visible light image data, infrared image data, local temperature distribution map, and derived key electrical feature data, providing a regular and effective data foundation for subsequent anomaly diagnosis and data fusion.

[0066] Step 102: Use a pre-set spatiotemporal structure pyramid network to perform anomaly diagnosis on key multimodal data.

[0067] In this embodiment of the invention, key multimodal data is input into a pre-set spatiotemporal structure pyramid network. This network has been pre-trained based on multimodal sample data of power equipment and can be directly used for anomaly diagnosis and localization. After receiving the key multimodal data, the network synchronously calls its internal spatial pyramid sub-network, temporal pyramid sub-network, and spatiotemporal fusion processing sub-network to work together. The spatial pyramid sub-network extracts multi-scale spatial fusion features from the input visible light image data, infrared image data, and local temperature distribution map. The temporal pyramid sub-network extracts temporal dynamic fusion features from the numerical features derived from the local temperature distribution map and key electrical features such as voltage and current. The spatiotemporal fusion processing sub-network performs cross-dimensional feature fusion and comprehensive analysis of the multi-scale spatial fusion features and temporal dynamic fusion features to identify whether there are abnormal operating conditions in the key multimodal data and outputs anomaly diagnosis results including the anomaly type and the location of the anomaly area, thus completing the preliminary judgment of the state of the target power equipment.

[0068] Step 103: Extract target data from key multimodal data based on the localization results in the anomaly diagnosis results, and determine data acquisition instructions that match the anomaly type in the anomaly diagnosis results.

[0069] In this embodiment of the invention, based on the anomaly diagnosis results, firstly, according to the anomaly area location information, visible light image data, infrared image data, local temperature distribution data, and key electrical characteristic data within the corresponding spatial location and time window are extracted from the key multimodal data to form target data focusing on anomaly-related information; subsequently, based on the predefined correspondence between anomaly types and acquisition parameters, an appropriate acquisition configuration is matched for the current anomaly type, and a data acquisition instruction containing parameters such as acquisition sensor type, acquisition area, sampling frequency, and data accuracy is generated.

[0070] Step 104: Control the data acquisition device that matches the data acquisition command to acquire data in the area corresponding to the positioning result, and obtain valid detection data.

[0071] In this embodiment of the invention, according to the generated data acquisition instruction, a data acquisition device (such as the corresponding sensor module carried by the inspection robot) that matches the parameters specified in the instruction is invoked, and the device moves to the area corresponding to the abnormal diagnosis result. Targeted data acquisition is performed according to the acquisition range, sampling frequency and data accuracy specified in the instruction to obtain supplementary data for that area. Subsequently, the acquired supplementary data is subjected to basic validity verification, and incomplete or incorrectly formatted invalid data is removed, ultimately obtaining valid detection data for the abnormal area.

[0072] Step 105: Map the target data and valid detection data to a unified latent space.

[0073] In this embodiment of the invention, firstly, modality-adaptive feature encoding is performed on the target data and effective detection data to generate initial latent vectors; then, dimension normalization processing is performed on all initial latent vectors to obtain intermediate latent vectors with consistent dimensions; finally, each intermediate latent vector is mapped to the same common feature space to complete the feature alignment of heterogeneous data and construct a unified latent space, providing a unified feature representation basis for multimodal data fusion modeling.

[0074] Step 106: Use a pre-set graph neural network to fuse and model the data in the unified latent space to obtain an anomaly detection fusion model, and use the anomaly detection fusion model to perform power equipment anomaly diagnosis and location, generating diagnosis and location results.

[0075] In this embodiment of the invention, the intermediate latent vectors in the constructed unified latent space are organized into graph structure data, with features from different modalities and sources as graph nodes. The edge weights between nodes are assigned based on spatiotemporal correlation and modal correlation. Then, the graph structure data is input into a pre-set graph neural network. The cross-modal fusion and correlation enhancement of node features are completed through the network's message passing mechanism. Then, the modeling is completed through classification and regression branches. Finally, an anomaly detection fusion model with the ability to classify abnormal operating conditions and locate abnormal areas is obtained. The anomaly detection fusion model is used to perform power equipment anomaly diagnosis and location, and generate diagnosis and location results.

[0076] Target power equipment area: The power equipment to be monitored and its surrounding area, such as the physical space containing transformers, insulators, transmission line joints, etc. within a substation; Multimodal data: Multiple types of data characterizing the state of power equipment acquired through different sensors or acquisition methods, including visible light image data, infrared image data, satellite remote sensing surface temperature data, voltage data, current data, partial discharge ultrasonic data, vibration data, etc.; Key multimodal data: A structured, high-quality dataset obtained after preprocessing the raw multimodal data (noise removal, format standardization, and extraction of effective features), used for subsequent analysis; Preset spatiotemporal context. Structural Pyramid Network: A pre-trained deep learning model comprising a spatial pyramid sub-network, a temporal pyramid sub-network, and a spatiotemporal fusion processing sub-network, used to extract spatiotemporal features from multimodal data and perform preliminary anomaly diagnosis; Anomaly Diagnosis Results: Conclusions output by the spatiotemporal structural pyramid network after processing key multimodal data, including anomaly type, severity, location information, and confidence level; Location Results: Information identifying the specific location, spatial coordinates, or area of ​​the suspected anomaly on the power equipment in the anomaly diagnosis results; Target Data: Based on the location results, multimodal data extracted from key multimodal data that is associated with the suspected anomaly area and time window. Modal data subset; Data acquisition instructions: Supplementary acquisition instructions generated based on the anomaly type in the anomaly diagnosis results, including parameters such as the type of sensor to be called, acquisition area, sampling frequency, and data accuracy; Data acquisition equipment: Sensors or devices used to supplement the acquired data, which can be mounted on inspection robots or integrated into power equipment, such as partial discharge ultrasonic sensors, hyperspectral imagers, vibration sensors, and temperature sensors; Valid detection data: High-quality supplementary data without format errors or missing data obtained after supplementary acquisition according to the data acquisition instructions and validity verification; Unified latent space: A low-dimensional common feature space constructed through the encoder network. Data from different modalities can be encoded and mapped to feature vectors of the same dimension within the space, achieving a unified representation of heterogeneous data; Pre-built graph neural network: a pre-constructed graph convolutional neural network used to fuse multimodal data features and learn the correlation between different modalities; Anomaly detection fusion model: based on data in a unified latent space, the final model obtained after graph neural network fusion modeling can output the anomaly diagnosis and anomaly location results of power equipment; Anomaly diagnosis: the process of identifying whether the operating status of power equipment is abnormal and judging the anomaly type and severity; Anomaly location: the process of determining the specific location, spatial position, or regional range of the anomaly occurring in power equipment.

[0077] Example 2:

[0078] This invention provides a multi-scale, multi-modal data fusion method for power scenarios, comprising:

[0079] Step 201: Collect multimodal data within the target power equipment area and preprocess it to obtain key multimodal data.

[0080] In this embodiment of the invention, an inspection robot is used to collect multimodal data in real time within the target power equipment area using integrated sensors. The multimodal data specifically includes visible light image data, infrared image data, satellite remote sensing surface temperature data, voltage data, and current data. During the acquisition process, all data are collected synchronously, accompanied by timestamps and location information (such as GPS coordinates) to ensure spatiotemporal alignment. Specifically, visible light image data is captured by a high-definition camera mounted on the inspection robot, capturing information such as the appearance and surrounding environment within the target power equipment area; infrared image data is obtained from an infrared thermal imager to acquire thermal distribution data of the equipment, used to detect overheating anomalies; satellite remote sensing surface temperature data is obtained from a satellite data source (such as receiving data via a wireless network) to assist in environmental analysis; voltage data is measured by voltage sensors (such as voltage transformers (PTs) or voltage divider circuits) to measure the power line voltage value, and current data is measured by current sensors (such as current transformers (CTs) or Hall effect current sensors, supporting non-destructive measurement) to measure the power line current value. Both are used together to monitor the operating status of the power equipment.

[0081] After data collection, lightweight computing nodes integrated into the inspection robot are used to independently preprocess each data type to eliminate noise, standardize the format, and extract preliminary features, ultimately obtaining key multimodal data. The preprocessing details for each data type are as follows:

[0082] Preprocessing of visible light image data: First, image enhancement is performed by adjusting brightness, contrast, and color balance to improve image quality. Second, noise reduction is performed by applying Gaussian filtering or median filtering to eliminate random noise in the image. Next, resizing and cropping are performed to scale the image to a uniform size and focus on key equipment areas (such as transformers and insulators). Finally, feature extraction is performed using lightweight computer vision algorithms (such as edge detection or Haar features) to identify key equipment areas, thus completing the preprocessing of visible light image data.

[0083] Preprocessing of infrared image data: First, temperature calibration is performed to correct temperature values ​​based on environmental parameters (such as humidity and distance) to ensure the accuracy of temperature data; second, hot spot detection is performed to identify overheated areas and mark temperature anomalies through threshold segmentation and clustering algorithms (such as K-means); finally, image registration is performed to align infrared image data with visible light image data to facilitate subsequent multimodal data fusion, thus completing the preprocessing of infrared image data.

[0084] Preprocessing of satellite remote sensing land surface temperature data: First, data parsing is performed to extract land surface temperature values ​​from the satellite data stream and convert them into a standard format (e.g., degrees Celsius); second, spatial matching is performed to match the temperature data with the location coordinates of the inspection robot to generate a local temperature distribution map; finally, normalization is performed to normalize the temperature values ​​to a uniform range (e.g., 0-1) to eliminate the influence of dimensions, thus completing the preprocessing of the satellite remote sensing land surface temperature data.

[0085] Preprocessing of voltage and current data: First, filtering and smoothing are performed, using moving averages or low-pass filters to remove high-frequency noise and transient interference from the data; second, outlier detection is performed, using statistical methods (such as Z-score) to identify and remove abnormal readings; finally, feature calculation is performed, deriving key electrical features such as power, power factor, and harmonic content from the processed data, thus completing the preprocessing of voltage and current data.

[0086] After the above independent preprocessing, the obtained key multimodal data includes regularized visible light image data, infrared image data, local temperature distribution maps, standardized surface temperature data, and derived key electrical characteristic data.

[0087] Furthermore, the spatiotemporal structure pyramid network includes a spatial pyramid subnetwork, a temporal pyramid subnetwork, and a spatiotemporal fusion processing subnetwork.

[0088] Spatial Pyramid Sub-network: A sub-module of the spatiotemporal structure pyramid network, used to extract multi-scale spatial features from image-like data and capture spatial information at different levels of power equipment; Temporal Pyramid Sub-network: A sub-module of the spatiotemporal structure pyramid network, used to extract multi-timescale dynamic features from time-series data and capture patterns of power equipment state changes over time; Spatiotemporal Fusion Processing Sub-network: A sub-module of the spatiotemporal structure pyramid network, used to fuse the multi-scale spatial fusion features output by the spatial pyramid sub-network with the temporal dynamic fusion features output by the temporal pyramid sub-network, achieving spatiotemporal feature interaction and outputting preliminary anomaly diagnosis results; Multi-scale Spatial Fusion Features: Features output by the spatial pyramid sub-network, fusing image information from different spatial levels (local details, component level, equipment level) of power equipment to characterize the spatial state features of the equipment; Temporal Dynamic Fusion Features: Features output by the temporal pyramid sub-network, fusing state change information of power equipment at different time scales (instantaneous, short-term, long-term) to characterize the temporal dynamic features of the equipment.

[0089] Step 202: Use the spatial pyramid sub-network to extract features from key multimodal data to obtain multi-scale spatial fusion features.

[0090] Furthermore, the spatial pyramid sub-network includes a convolutional neural network backbone, multi-level pooling units, and upsampling and concatenation layers. The multi-level pooling units include multiple cascaded pooling layers. Step 202 may include the following sub-steps:

[0091] S11. Use the backbone of a convolutional neural network to extract features from the visible light image data and infrared image data in the key multimodal data to obtain the initial feature map.

[0092] S12. Multiple cascaded pooling layers are used to perform multi-scale pooling on the initial feature map to obtain multiple sub-feature maps.

[0093] S13. Upsample multiple sub-feature maps according to a preset size using an upsampling and stitching layer, and stitch the upsampling results together to obtain multi-scale spatial fusion features.

[0094] In the embodiments of the present invention, please refer to Figure 2 This step aims to simultaneously capture and diagnose anomalous features at different spatial scales. Basic feature extraction: The convolutional neural network backbone uses a shared weight structure. Taking preprocessed visible light and infrared image data as input, it extracts low-level texture, edge details, and high-level semantic features through multiple convolutional operations, generating an initial, high-resolution feature map. This initial feature map fully preserves the basic spatial information of the image data, while the shared weight mechanism ensures consistency between visible light and infrared image feature extraction, providing a unified feature foundation for multi-scale pooling. Multi-scale spatial feature construction: Multiple cascaded pooling layers employ average pooling to progressively downsample the initial feature map, gradually reducing its size and generating a set of sub-feature maps with decreasing resolution but increasing receptive field, forming a "spatial pyramid" structure. Different levels of sub-feature maps correspond to different scales. Feature capture: Level 1 (high resolution) focuses on subtle features, such as tiny cracks or dirt on insulators; Level 2 (medium resolution) focuses on component-level features, such as the uniformity of temperature distribution throughout the insulator string; Level 3 (low resolution) focuses on equipment-level or region-level features, such as the thermal pattern of the entire transformer enclosure or its contrast with the surrounding environment, achieving hierarchical extraction of spatial information at different granularities; Feature fusion: By upsampling, sub-feature maps of different resolutions are restored to the same preset size, eliminating scale differences; then, by channel-dimensional splicing or weighted fusion, the upsampled sub-feature maps are integrated to finally generate a fused feature representation rich in multi-scale spatial information, i.e., multi-scale spatial fusion feature; This feature simultaneously includes details of local defects in the equipment, component-level state distribution, and equipment / region-level operational characterization, achieving complementary integration of spatial information at different scales.

[0095] Convolutional Neural Network Backbone: The foundational module of the spatial pyramid subnetwork, a weighted convolutional neural network used to extract initial feature maps from visible light and infrared image data; Multi-level Pooling Unit: A module of the spatial pyramid subnetwork containing multiple cascaded pooling layers used to downsample the initial feature maps at multiple scales, generating sub-feature maps of different resolutions; Upsampling and Stitching Layer: A module of the spatial pyramid subnetwork used to upsample sub-feature maps of different resolutions to the same size and stitch them together to obtain multi-scale spatial fusion features; Multiple Cascaded Pooling Layers: Components of the multi-level pooling unit, using different pooling kernel sizes and strides to downsample the initial feature maps step by step, generating sub-feature maps with increasing receptive fields and decreasing resolutions; Initial Feature Map: The high-resolution basic feature representation extracted from visible light and infrared image data, output by the convolutional neural network backbone; Sub-feature Maps: Feature maps of different resolutions obtained after downsampling the initial feature map by the multi-level pooling units, corresponding to different spatial scales.

[0096] Step 203: Use the temporal pyramid sub-network to extract features from key multimodal data to obtain temporal dynamic fusion features.

[0097] Furthermore, the temporal pyramid sub-network includes a numerical feature extraction unit, a multi-timescale temporal convolution unit, and a multi-timescale feature fusion unit. Step 203 may include the following sub-steps:

[0098] S21. Numerical features are extracted from visible light image data, infrared image data, and local temperature distribution map in key multimodal data using the numerical feature extraction unit to obtain numerical features.

[0099] S22. Use multi-timescale temporal convolutional units to perform multi-timescale dynamic feature processing on voltage data, current data and numerical features in key multimodal data to obtain dynamic features corresponding to different time scales.

[0100] S23. Use a multi-timescale feature fusion unit to fuse the dynamic features to obtain time-dynamic fused features.

[0101] In the embodiments of the present invention, please refer to Figure 3This step aims to analyze the patterns of feature changes over time, identify instantaneous, short-term, or long-term abnormal trends, and for non-time-series image and temperature data, extract quantifiable numerical features (such as average gray value, regional temperature mean / variance, pixel statistical features of key equipment areas, etc.) from preprocessed visible light image data, infrared image data, and local temperature distribution maps. This converts unstructured image and temperature data into structured numerical forms, providing feature inputs compatible with electrical data for subsequent time-series processing. Voltage data, current data, and the numerical features extracted in S21 are used as unified inputs within a preset time window based on their temporal changes. Subsequently, based on a temporal convolutional network (or lightweight recurrent neural network), the input is processed by adjusting the kernel size, stride, or using different sampling rates. The data undergoes multi-scale time-series processing to capture dynamic information at different time scales: short time scales (using small convolution kernels / short strides) can capture sudden anomalies such as instantaneous shocks and voltage drops; medium time scales can identify overload or heating trends lasting from several seconds to several minutes; long time scales (using large convolution kernels / long strides) can analyze slow degradation processes measured in hours, such as the gradual decline in equipment performance, ultimately yielding dynamic features corresponding to different time scales. The dynamic features extracted from different time scales are fused to form a feature representation that comprehensively depicts the time dynamic changes during the operation of power equipment, namely, the time dynamic fusion feature. This feature simultaneously includes time-series information on instantaneous abrupt changes, short-term trends, and long-term degradation, providing complete time-dimensional feature support for subsequent spatiotemporal fusion and anomaly diagnosis.

[0102] Numerical Feature Extraction Unit: A module of the temporal pyramid sub-network, used to convert non-temporal image data and local temperature distribution maps into structured numerical features, achieving temporal adaptation of data; Multi-timescale Temporal Convolution Unit: A module of the temporal pyramid sub-network, based on a temporal convolutional network, capturing dynamic information at different time scales by adjusting the convolutional kernel size, stride, or sampling rate; Multi-timescale Feature Fusion Unit: A module of the temporal pyramid sub-network, used to fuse dynamic features at different time scales to obtain temporal dynamic fusion features; Local Temperature Distribution Map: Structured data representing the surface temperature distribution within the inspection area, generated by spatial matching of satellite remote sensing surface temperature data; Numerical Features: Structured numerical features extracted from non-temporal image data and local temperature distribution maps, compatible with electrical time-series data, such as average grayscale value of the image, temperature mean / variance, etc.; Dynamic Features Corresponding to Different Time Scales: Dynamic features of power equipment status output by the multi-timescale temporal convolution unit, corresponding to instantaneous, short-term, and long-term time windows respectively.

[0103] Step 204: Use the spatiotemporal fusion processing sub-network to perform spatiotemporal feature fusion and anomaly diagnosis on multi-scale spatial fusion features and temporal dynamic fusion features to obtain anomaly diagnosis results.

[0104] In this embodiment of the invention, the obtained multi-scale spatial fusion features and temporal dynamic fusion features are input into the spatiotemporal fusion processing sub-network for deep spatiotemporal feature fusion and interaction. The network introduces a cross-attention mechanism to learn the correlation logic of "when to pay attention to where": for example, when the temporal dynamic fusion features detect timing anomalies such as current spikes, the attention mechanism will automatically strengthen the region of the corresponding electrical connection point in the multi-scale spatial fusion features, focusing on whether the region simultaneously shows signs of overheating or visible photoelectric arc emission, thereby achieving enhanced correlation of spatiotemporal features.

[0105] Next, the multi-scale spatial fusion features after cross-attention interaction are concatenated with the temporal dynamic fusion features to obtain the target fusion features. Then, the target fusion features are compressed and dimensionality reduced through a fully connected layer (or convolutional layer) to remove redundant information and extract the most discriminative spatiotemporal joint features.

[0106] Finally, the dimensionality-reduced spatiotemporal joint features are input into the final classifier (such as a fully connected layer and a softmax layer) or regressor to complete the anomaly diagnosis and output an anomaly diagnosis result containing the following information:

[0107] Abnormal type: such as "insulator flashover", "conductor joint overheating", "internal winding fault" or "normal", etc.;

[0108] Severity level: such as "mild", "moderate", "severe", etc.;

[0109] Location information: By combining the high-resolution output of the spatial pyramid subnetwork, the specific location of the anomaly in the device can be determined;

[0110] Confidence level: The degree to which the network is confident about the diagnostic result; at the same time, the diagnostic result, confidence level, and the timestamp and location information attached to the data are packaged to form a complete abnormal diagnostic result.

[0111] It is worth mentioning that the improved spatiotemporal pyramid network in this invention achieves deep spatiotemporal feature mining of multimodal power data through a dual-branch collaborative design. The spatial pyramid sub-network abandons the conventional spatial pyramid method of using only a single pooling branch or direct stitching of scale features. Instead, it uses a convolutional neural network backbone as the basic image feature extraction unit, combined with multi-level cascaded pooling units to generate multi-scale sub-feature maps. After upsampling and stitching layers unify the features of each scale to the same size, fusion is completed. This structural improvement solves the problems of scale feature dimension mismatch and difficulty in simultaneously addressing local detail features and global contour features in conventional spatial pyramid networks. It can accurately extract defect and state features at different scales in visible light and infrared images of power equipment, significantly improving the completeness and representational ability of spatial features. The temporal pyramid sub-network breaks through the conventional time-series feature processing that only targets a single electrical time series. Overcoming the limitations of data modeling, this approach first transforms non-time-series data such as images and temperatures into numerical features, then combines them with time-series data of voltage and current. Multi-timescale temporal convolutional units are used to mine dynamic change patterns across different time spans. Finally, a multi-timescale feature fusion unit integrates these features to obtain temporal dynamic fusion features. This effectively compensates for the shortcomings of conventional time-series networks, which cannot fuse multi-modal temporal correlation information and can only capture changes in operating conditions at a single time scale. It can comprehensively perceive the temporal characteristics of short-term operational fluctuations and long-term performance degradation of power equipment. The features extracted by the two sub-networks are fused across dimensions through a spatiotemporal fusion processing sub-network, enabling the network to simultaneously consider multi-scale spatial details and full-cycle temporal dynamic characteristics. This structurally solves the problem of fragmented and insufficiently fused spatiotemporal features in multi-source heterogeneous power data, providing a highly robust feature foundation for anomaly diagnosis and localization, and further improving the accuracy and reliability of anomaly diagnosis.

[0112] Step 205: Extract target data from key multimodal data based on the localization results in the anomaly diagnosis results, and determine data acquisition instructions that match the anomaly type in the anomaly diagnosis results.

[0113] In this embodiment of the invention, firstly, based on the output abnormal diagnosis results, the location information contained therein (such as the specific location of the device, the spatial coordinate range, and the corresponding time window) is analyzed. From the key multimodal data obtained by preprocessing, visible light image data, infrared image data, local temperature distribution map, key electrical feature data, etc., corresponding to the spatial location and the corresponding time interval are screened and extracted. Redundant data in irrelevant areas are eliminated to form target data focusing on abnormal events, ensuring that subsequent analysis only revolves around the relevant data of suspected abnormal areas, thereby improving data processing efficiency and analysis targeting.

[0114] Next, the abnormality types in the abnormality diagnosis results (such as partial discharge, insulation aging, equipment overheating, etc.) are analyzed. Based on the preset correspondence between abnormality types and acquisition configurations, the appropriate acquisition parameters are matched for the current abnormality, and clear and executable data acquisition instructions are generated. The instructions must include key parameters such as the type of sensor to be called (such as partial discharge ultrasonic sensor, hyperspectral imager, power equipment built-in vibration / temperature sensor, etc.), acquisition area range, sampling frequency, data accuracy requirements, and acquisition duration, so as to provide a direct basis for calling the matching data acquisition equipment and carrying out targeted supplementary acquisition.

[0115] Step 206: Control the data acquisition device that matches the data acquisition command to acquire data in the area corresponding to the positioning result, and obtain valid detection data.

[0116] In this embodiment of the invention, firstly, the data acquisition device can be mounted on an inspection robot, such as a partial discharge ultrasonic sensor or a hyperspectral imager (used to detect insulation aging of equipment); or it can be a sensor built into the power equipment, such as a vibration sensor or a temperature sensor, etc. The device type matches the anomaly type specified in the data acquisition instruction generated in step 205.

[0117] Based on the anomaly diagnosis results from step 204, this step focuses on real-time data acquisition of the target, such as prioritizing scanning transformers with abnormal temperatures. During data acquisition, the inspection robot moves to the positioning area corresponding to the anomaly diagnosis results and calls the matching data acquisition device to perform targeted data acquisition of the area according to the acquisition parameters specified in the data acquisition instructions (such as sensor type, sampling frequency, and data accuracy). After the acquisition is completed, the validity of the acquired supplementary data is verified, and invalid data with incorrect format or incomplete data is removed, ultimately obtaining valid detection data.

[0118] Step 207: Map the target data and valid detection data to a unified latent space.

[0119] Furthermore, step 207 may include the following sub-steps:

[0120] S31. Encode the target data and the effective detection data to obtain multiple initial hidden vectors;

[0121] S32. Perform dimensionality reduction on all initial latent vectors to obtain intermediate latent vectors;

[0122] S33. Map all intermediate latent vectors to the same feature space to construct a unified latent space.

[0123] In this embodiment of the invention, the goal of this step is to create a common, low-dimensional mathematical space (latent space) so that all different types of data and new data can be represented and compared in this space using a unified "language" (i.e., feature vectors). A dedicated encoder is configured for each type of key multimodal data (such as image features, infrared thermal features, and electrical time-series features). The encoder is usually a lightweight neural network and is matched with the data type: the encoder for image data (such as visible light image data, infrared image data, and local temperature distribution maps) is a convolutional neural network, and the encoder for numerical data (such as key electrical features and numerical features in effective detection data) is a fully connected network. These encoders are trained collaboratively and can convert the original modal data they are responsible for into semantically meaningful latent vectors. In this step, the target data extracted in step 205 and the effective detection data obtained in step 206 are input into their respective pre-trained encoders of the corresponding type to obtain initial latent vectors of a preset dimension. All initial latent vectors have the same dimension to ensure that they can be mapped to the same latent space later. At the same time, newly detected effective detection data are mapped in real time using the same encoder group as the target data to ensure that the new and old data are compared and analyzed on the same benchmark, maintaining the consistency of the model's judgment criteria. To highlight effective information and reduce the amount of subsequent computation, the initial latent vectors obtained in S31 are dimensionality reduced to obtain low-dimensional intermediate latent vectors. Latent vectors; the dimension of intermediate latent vectors is smaller than the preset dimension of the initial latent vectors, belonging to low-dimensional feature representation. They can compress data dimensions while retaining key semantic information, thereby improving the efficiency of subsequent processing. The dimensionality-reduced intermediate latent vectors are mapped to the same common feature space to construct a unified latent space. In this latent space, the geometric distance of data points can reflect the similarity of their device operating states. For example, all device data points in the "normal" state will cluster in the same area, while data points in various "abnormal" states (such as overheating and partial discharge) will be distributed in other specific areas, providing a unified feature basis for the construction of subsequent anomaly detection fusion models.

[0124] Initial latent vector: A feature vector of a preset dimension obtained after the target data and effective detection data are processed by a dedicated encoder. The initial latent vectors of all modal data have the same dimension. Dimensionality reduction operation: The process of compressing the initial latent vector to reduce the feature dimension, highlight effective information, and reduce the amount of computation. Intermediate latent vector: A low-dimensional feature vector obtained after the initial latent vector is reduced, which is the basis for the data representation in the unified latent space.

[0125] It is worth mentioning that the latent space mapping step of this invention is not a traditional single-encoding mapping mode, but rather forms a deep synergy with the improved spatiotemporal structure pyramid network. Traditional latent space mapping often adopts a single encoding and simple splicing method, which cannot break the semantic barriers of heterogeneous data and is difficult to filter redundant noise, resulting in insufficient multimodal data fusion and insufficient diagnostic accuracy. In this invention, the spatiotemporal structure pyramid network first selects high-quality, highly correlated target data and effective detection data through multi-scale spatiotemporal feature extraction and preliminary diagnosis, providing a reliable foundation for latent space mapping. Then, through progressive operations of encoding, dimensionality reduction, and unified mapping, it specifically adapts to multimodal data features, breaks down heterogeneous data barriers to achieve semantic alignment, and effectively solves the problems of feature loss and fusion disconnect in traditional mapping. At the same time, it complements the feature extraction of the spatiotemporal structure pyramid network, giving full play to the synergistic value of multimodal data, greatly improving the efficiency and quality of multimodal data fusion, further enhancing the accuracy and reliability of anomaly diagnosis, and effectively making up for the shortcomings of traditional methods in multi-source heterogeneous data fusion and anomaly diagnosis accuracy, providing a solid guarantee for the safe and stable operation of power equipment.

[0126] Step 208: Use a pre-set graph neural network to fuse and model the data in the unified latent space to obtain an anomaly detection fusion model, and use the anomaly detection fusion model to perform power equipment anomaly diagnosis and location, generating diagnosis and location results.

[0127] Furthermore, step 208 may include the following sub-steps:

[0128] S41. Construct a modality relationship graph corresponding to each modality data in the unified latent space; the nodes of the modality relationship graph are each modality, and the node features are the intermediate latent vectors of the corresponding modality in the unified latent space;

[0129] S42. Calculate the cosine similarity between nodes in the intermodal relationship graph and perform exponential normalization. Use the exponential normalization result as the edge weight.

[0130] S43. Using a pre-built graph neural network, based on the inter-modal relationship graph and edge weights, graph convolution fusion processing is performed on the intermediate latent vectors in the unified latent space to obtain an anomaly detection fusion model.

[0131] S44. Perform power equipment anomaly diagnosis and location through anomaly detection fusion model, and generate diagnosis and location results.

[0132] In this embodiment of the invention, this step utilizes a graph neural network for model construction, which can deeply integrate multi-source heterogeneous power equipment status data, accurately model the physical topology of the power system, and capture fault propagation and correlation effects. Using the unified latent space constructed in step 207, each mode corresponding to the target data and effective detection data (such as visible light image mode, infrared image mode, electrical timing mode, local temperature distribution mode, supplementary mode of effective detection data, etc.) is used as a node, and the intermediate latent vector corresponding to each mode is used as the node feature to construct an inter-modal relationship graph. This relationship graph is used to characterize the correlation structure between different modal data. First, the intermediate latent vectors corresponding to any two nodes (modalities) are calculated. and Cosine similarity between The formula is as follows:

[0133]

[0134] The cosine similarity was then exponentially normalized to obtain the node... With nodes Edge weights between The formula is as follows:

[0135]

[0136] In the formula: Indicates the first The feature vectors of each modality in the unified latent space; Indicates the first The feature vectors of each modality in the unified latent space; Indicates the first The feature vectors of each modality in the unified latent space; Indicates the number of modes; For the first The modality and the first Cosine similarity of each modality; For nodes With nodes The edge weights are used to characterize the correlation strength between two modal features.

[0137] The constructed intermodal relationship graph and edge weights are input into a pre-set graph neural network. The network uses graph convolutional layers to perform message passing and feature updates on the intermediate latent vectors of each node (modality) based on edge weights, achieving deep fusion of multimodal data. During the fusion process, the graph neural network can learn the correlation patterns between different modal data, capture the propagation path and correlation effects of faults in the operation of power equipment, and finally train an anomaly detection fusion model with both anomaly diagnosis (such as anomaly type identification) and anomaly localization (such as identification of specific fault locations in equipment). The anomaly detection fusion model is then used to perform anomaly diagnosis and localization of power equipment, generating diagnosis and localization results.

[0138] Intermodal Relationship Graph: A topological graph constructed with different modal data as nodes and the corresponding intermediate latent vectors of the modal as node features, used to represent the association relationships between multimodal data; Node: The constituent unit of the intermodal relationship graph, corresponding to a type of multimodal data of power equipment; Node Feature: The intermediate latent vector corresponding to the node in the intermodal relationship graph, representing the feature representation of the modal data in a unified latent space; Cosine Similarity: An index that measures the similarity of the directions of two vectors, used here to calculate the association strength of intermediate latent vectors of different modalities; Exponential Normalization: The process of performing an exponential operation on the cosine similarity and then summing and normalizing it according to the number of modalities, used to obtain the edge weights of the intermodal relationship graph; Edge Weight: The weight value of the edge connecting two nodes in the intermodal relationship graph, obtained by exponential normalization of cosine similarity, representing the association strength between two modal data; Graph Convolutional Fusion Processing: Using the convolutional layers of the graph neural network, based on the topological structure and edge weights of the intermodal relationship graph, message passing and updating of node features are performed to achieve deep fusion of multimodal data.

[0139] This invention addresses the challenges of multi-source heterogeneous data fusion and insufficient anomaly diagnosis accuracy in intelligent operation and maintenance scenarios for power equipment. First, it preprocesses multimodal sensing data from the target equipment area to form standardized key multimodal data. The core of this approach is a pre-built spatiotemporal pyramid network for initial anomaly diagnosis. This network uses a spatial pyramid sub-network to extract hierarchical features from visible light and infrared images using a multi-scale pooling strategy, taking into account both component-level defect details and overall operational status representation. Simultaneously, a temporal pyramid sub-network uses multi-scale temporal convolution to capture dynamic changes in voltage, current, and derived features, covering short-term operating condition fluctuations and long-term degradation trends. Finally, a spatiotemporal fusion processing sub-network achieves cross-correlation between spatial and temporal features, overcoming the technical bottlenecks of existing single-scale feature extraction methods, such as loss of spatiotemporal dimension information and inability to simultaneously represent multi-scale states. This provides accurate positioning and type guidance for subsequent targeted data acquisition. Based on the preliminary diagnostic results, this invention extracts target data from key multimodal data and matches it with corresponding data acquisition instructions. This drives specialized equipment to perform supplementary detection on suspected anomaly areas, obtaining valid detection data. Subsequently, the target data and valid detection data are mapped to a unified latent space to achieve semantic alignment of heterogeneous data. Finally, a pre-built graph neural network is used to mine deep correlation features between multimodal data, completing fusion modeling and obtaining a detection model with both anomaly diagnosis and localization capabilities. Compared to traditional multimodal fusion schemes, this invention uses a spatiotemporal structure pyramid network as its core support. Through hierarchical extraction and cross-fusion of multi-scale spatiotemporal features, it significantly improves the feature utilization rate of multi-source heterogeneous data and strengthens the semantic correlation between cross-modal data. Simultaneously, targeted supplementary acquisition avoids redundant acquisition of invalid data, further ensuring the integrity of key information. The final anomaly detection model significantly improves the diagnostic accuracy and localization precision of typical defects in power equipment. Its robustness to complex field conditions such as noise interference and partial data loss is also effectively enhanced, providing a reliable guarantee for the safe and stable operation and maintenance of power equipment.

[0140] Example 3:

[0141] Please see Figure 4 , Figure 4 This is a structural block diagram of a multi-scale, multi-modal data fusion system for power scenarios provided in Embodiment 3 of the present invention.

[0142] This invention provides a multi-scale, multi-modal data fusion system for power scenarios, comprising:

[0143] The data acquisition module 401 is used to collect multimodal data within the target power equipment area through the inspection robot and perform preprocessing to obtain key multimodal data;

[0144] The spatiotemporal structure pyramid network module 402 is used to perform anomaly diagnosis on key multimodal data using a pre-set spatiotemporal structure pyramid network.

[0145] The data extraction and processing module 403 is used to extract target data from key multimodal data based on the location results in the anomaly diagnosis results, and to determine the data acquisition instructions that match the anomaly type in the anomaly diagnosis results.

[0146] The data acquisition device module 404 is used to control the data acquisition device that matches the data acquisition command to acquire data in the area corresponding to the positioning result, so as to obtain valid detection data.

[0147] Encoder network module 405 is used to map target data and valid detection data to a unified latent space;

[0148] The graph neural network module 406 is used to fuse and model data in a unified latent space using a pre-set graph neural network to obtain an anomaly detection fusion model, and to perform power equipment anomaly diagnosis and location through the anomaly detection fusion model to generate diagnosis and location results.

[0149] Since the above is a system corresponding to a multi-scale multimodal data fusion method for power scenarios, its implementation principle is consistent with that of a multi-scale multimodal data fusion method for power scenarios. For the sake of convenience and brevity, those skilled in the art can clearly understand that the specific working process of the system and modules described above can be referred to the corresponding process in the aforementioned method embodiments, and will not be repeated here.

[0150] This invention utilizes an inspection robot to collect multimodal data within the target power equipment area, and preprocesses the multimodal data using lightweight computing nodes integrated into the inspection robot to obtain key multimodal data. A spatiotemporal pyramid network is used for anomaly diagnosis of the key multimodal data. A data extraction and processing module extracts target data from the key multimodal data based on the location results in the anomaly diagnosis results, and determines data acquisition commands matching the anomaly types in the anomaly diagnosis results. The spatiotemporal pyramid network includes a spatial pyramid sub-network, a temporal pyramid sub-network, and a spatiotemporal fusion processing sub-network. Data acquisition devices matching the data acquisition commands are controlled to collect data from the area corresponding to the location results, obtaining effective detection data. An encoder network maps the target data and effective detection data to a unified latent space. A graph neural network is used to fuse and model the data in the unified latent space, obtaining an anomaly detection fusion model. This anomaly detection fusion model is used for anomaly diagnosis and anomaly location. This invention realizes the cognitive logic of "macroscopic investigation → microscopic focus → comprehensive modeling," achieving efficient and accurate data fusion and analysis. The anomaly detection fusion model obtained by data fusion processing can effectively improve the consistency level of fault detection while significantly reducing fault response time. The spatiotemporal structure pyramid network analyzes the spatial and temporal (trends, sequences) characteristics of data at multiple scales and levels, thus facilitating the subsequent construction of anomaly detection fusion models capable of accurately diagnosing power equipment anomalies. This model effectively detects and diagnoses various anomalies, ranging from local to global and from instantaneous to continuous. The encoder network creates a low-dimensional latent space specifically optimized for anomaly patterns, where different types of anomalies form well-separable clustered distributions. Graph neural networks, a graph convolutional fusion method based on intermodal relationship graphs, effectively utilize the complex relationships between multimodal data, improving the accuracy and robustness of anomaly detection. In summary, the technical solution disclosed in this paper effectively addresses the shortcomings of existing multimodal data fusion technologies in power scenarios, particularly in areas such as unified representation of heterogeneous data, cross-modal semantic association modeling, and multi-scale spatiotemporal feature mining.

[0151] Example 4:

[0152] Please see Figure 5 , Figure 5 This is a structural block diagram of an electronic device provided in Embodiment 4 of the present invention.

[0153] An electronic device according to an embodiment of the present invention includes: a memory 501 and a processor 502. The memory 501 stores a computer program. When the computer program is executed by the processor 502, the processor 502 executes the multi-scale multimodal data fusion method for power scenarios as described in the above embodiment.

[0154] Memory 501 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Memory 501 has storage space 503 for program code 513 for performing any of the method steps described above. For example, storage space 503 for program code may include various program codes 513 for implementing the various steps in the methods described above. This program code can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, CDs, memory cards, or floppy disks. The program code may be compressed, for example, in a suitable form. When run by a computing processing device, this code causes the computing processing device to perform the various steps in the methods described above. This program code can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, CDs, memory cards, or floppy disks. The program code may be compressed, for example, in a suitable form. When this code is run by a computing device, it causes the device to perform the various steps in the multi-scale, multi-modal data fusion method for power scenarios described above.

[0155] Example 5:

[0156] Embodiment 5 of the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multi-scale, multi-modal data fusion method for power scenarios as described in the above embodiments.

[0157] Example 6:

[0158] Embodiment 6 of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, wherein when the program instructions are executed by a computer, the computer performs the multi-scale multimodal data fusion method for power scenarios as described in the above embodiments.

[0159] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0160] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0161] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0162] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0163] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0164] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-scale, multi-modal data fusion method for power scenarios, characterized in that, include: Collect multimodal data within the target power equipment area and preprocess it to obtain key multimodal data; Anomaly diagnosis is performed on the key multimodal data using a pre-set spatiotemporal structure pyramid network; Based on the localization results in the anomaly diagnosis results, target data is extracted from the key multimodal data, and data acquisition instructions that match the anomaly type in the anomaly diagnosis results are determined. The data acquisition device, which is matched with the data acquisition command, is controlled to acquire data in the area corresponding to the positioning result, thereby obtaining valid detection data. Map the target data and the effective detection data to a unified latent space; A pre-defined graph neural network is used to fuse and model the data in the unified latent space to obtain an anomaly detection fusion model. The anomaly detection fusion model is then used to perform anomaly diagnosis and location of power equipment, generating diagnosis and location results.

2. The multi-scale, multi-modal data fusion method for power scenarios according to claim 1, characterized in that, The spatiotemporal structure pyramid network includes a spatial pyramid sub-network, a temporal pyramid sub-network, and a spatiotemporal fusion processing sub-network. The anomaly diagnosis of the key multimodal data using the pre-set spatiotemporal structure pyramid network includes: The spatial pyramid subnetwork is used to extract features from the key multimodal data to obtain multi-scale spatial fusion features; The key multimodal data are used to extract features using the temporal pyramid subnetwork to obtain temporal dynamic fusion features; The spatiotemporal fusion processing subnetwork is used to perform spatiotemporal feature fusion and anomaly diagnosis on the multi-scale spatial fusion features and the temporal dynamic fusion features to obtain the anomaly diagnosis result.

3. The multi-scale, multi-modal data fusion method for power scenarios according to claim 2, characterized in that, The spatial pyramid sub-network includes a convolutional neural network backbone, multi-level pooling units, and upsampling and concatenation layers. The multi-level pooling units include multiple cascaded pooling layers. The feature extraction of the key multimodal data using the spatial pyramid sub-network to obtain multi-scale spatial fusion features includes: The visible light image data and infrared image data in the key multimodal data are extracted using the backbone of the convolutional neural network to obtain an initial feature map; The initial feature map is subjected to multi-scale pooling processing using the multiple cascaded pooling layers to obtain multiple sub-feature maps; The upsampling and stitching layer is used to upsample multiple sub-feature maps according to a preset size, and the upsampling results are stitched together to obtain multi-scale spatial fusion features.

4. The multi-scale, multi-modal data fusion method for power scenarios according to claim 2, characterized in that, The temporal pyramid sub-network includes a numerical feature extraction unit, a multi-timescale temporal convolution unit, and a multi-timescale feature fusion unit. The step of using the temporal pyramid sub-network to extract features from the key multimodal data to obtain time-dynamically fused features includes: The numerical feature extraction unit is used to extract numerical features from the visible light image data, infrared image data, and local temperature distribution map in the key multimodal data to obtain numerical features; The voltage data, current data and numerical features in the key multimodal data are processed using the multi-timescale temporal convolutional unit to obtain dynamic features corresponding to different time scales. The dynamic features are fused using the multi-timescale feature fusion unit to obtain time-dynamic fused features.

5. The multi-scale, multi-modal data fusion method for power scenarios according to any one of claims 1-4, characterized in that, The step of mapping the target data and the valid detection data to a unified latent space includes: The target data and the effective detection data are encoded to obtain multiple initial hidden vectors; Dimensionality reduction is performed on all the initial latent vectors to obtain intermediate latent vectors; All the intermediate latent vectors are mapped to the same feature space to construct a unified latent space.

6. The multi-scale, multi-modal data fusion method for power scenarios according to claim 5, characterized in that, The process involves fusing and modeling data in the unified latent space using a pre-set graph neural network to obtain an anomaly detection fusion model. This model is then used to perform anomaly diagnosis and location for power equipment, generating diagnosis and location results, including: Construct an intermodal relationship graph corresponding to each modality data in the unified latent space; the nodes of the intermodal relationship graph are each modality, and the node features are the intermediate latent vectors of the corresponding modality in the unified latent space; Calculate the cosine similarity between nodes in the intermodal relationship graph and perform exponential normalization, then use the exponential normalization result as the edge weight. Using a pre-built graph neural network, based on the inter-modal relationship graph and the edge weights, graph convolution fusion processing is performed on the intermediate latent vectors in the unified latent space to obtain the anomaly detection fusion model; The anomaly detection fusion model is used to perform anomaly diagnosis and location of power equipment, and generate diagnosis and location results.

7. A multi-scale, multi-modal data fusion system for power scenarios, characterized in that, include: The acquisition module is used to acquire multimodal data within the target power equipment area and perform preprocessing to obtain key multimodal data; The spatiotemporal structure pyramid network module is used to perform anomaly diagnosis on the key multimodal data using a pre-set spatiotemporal structure pyramid network. The data extraction and processing module is used to extract target data from the key multimodal data based on the location results in the anomaly diagnosis results, and to determine the data acquisition instructions that match the anomaly type in the anomaly diagnosis results. The data acquisition device module is used to control a data acquisition device that matches the data acquisition command to acquire data in the area corresponding to the positioning result, so as to obtain valid detection data. The encoder network module is used to map the target data and the valid detection data to a unified latent space; The graph neural network module is used to fuse and model the data in the unified latent space using a pre-set graph neural network to obtain an anomaly detection fusion model, and to perform power equipment anomaly diagnosis and location through the anomaly detection fusion model to generate diagnosis and location results.

8. An electronic device, characterized in that, The system includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the multi-scale, multi-modal data fusion method for power scenarios as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the multi-scale, multi-modal data fusion method for power scenarios as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, wherein when the program instructions are executed by a computer, the computer performs the multi-scale multimodal data fusion method for power scenarios as described in any one of claims 1-6.

Images