Substation intelligent inspection robot anomaly identification method and system based on multi-modal fusion and deep learning

Abstract

The invention belongs to the technical field of substation equipment intelligent inspection and anomaly recognition, and particularly relates to a substation intelligent inspection robot anomaly recognition method and system based on multi-modal fusion and deep learning. The objective of the invention is to solve the problems of single mode limitation, poor working condition adaptability and the like in substation equipment abnormity identification. The method comprises the following steps: synchronously acquiring multi-modal time sequence data through an inspection robot, and embedding a GPS position and a unique ID of equipment to complete scene marking; after preprocessing and time alignment, extracting depth features of each modal; in combination with the working condition parameters, outputting an adaptive weight by a dynamic weight fusion network, and generating a combined multi-modal feature; and through a space-time double-flow deep learning network, synchronously capturing space static characteristics and time dynamic characteristics, identifying an abnormal state, and positioning an abnormal position. The system comprises a data acquisition module, a preprocessing module, a feature extraction module and other core modules, supports online updating of the model, adapts to complex dynamic working conditions of the transformer substation, and improves the inspection efficiency and the anomaly recognition accuracy.

Description

Technical Field

[0001] This invention belongs to the field of intelligent inspection and anomaly recognition technology for substation equipment, and particularly relates to an anomaly recognition method and system for intelligent inspection robots in substations based on multimodal fusion and deep learning. Background Technology

[0002] As a core hub in the power generation, transmission, and distribution chain, the operating status of substations directly determines the security and stability of power supply. With the continuous expansion of the power grid and the increasing complexity of equipment types, traditional manual inspection methods are no longer sufficient to meet the demands of modern operation and maintenance. On the one hand, manual inspection relies on the experience of maintenance personnel, resulting in high labor intensity, low inspection efficiency, and susceptibility to problems such as missed or incorrect inspections due to harsh environments like high temperatures, heavy rain, and nighttime conditions, as well as subjective human factors. This makes it difficult to detect early potential anomalies in equipment in a timely manner. On the other hand, substation equipment anomalies are diverse (such as overheating, mechanical loosening, partial discharge, and abnormal noise), and the characteristics of different anomalies are scattered across multiple dimensions, including appearance, temperature, sound, and electrical signals. Manual inspection struggles to comprehensively capture multi-dimensional anomaly information.

[0003] To address the limitations of manual inspections, intelligent inspection robots are gradually being applied in substation scenarios. However, existing technologies still have significant shortcomings: First, most intelligent inspection systems rely on single-modal data for anomaly identification. Single-modal data can only capture local information and cannot achieve comprehensive coverage of different types of anomalies, resulting in insufficient completeness of anomaly identification. Second, some multimodal fusion solutions adopt a fixed-weight fusion strategy, ignoring the dynamic changes in substation operating conditions. Fixed weights cannot adapt to complex operating conditions, resulting in insufficient discriminative power of fused features. Third, existing anomaly identification models mostly focus on capturing single spatial or temporal features, making it difficult to consider both the static state of equipment (such as external damage and temperature distribution) and dynamic trends (such as continuous temperature increases and periodic discharge signals), resulting in low identification accuracy. At the same time, anomaly localization is mostly limited to equipment-level localization and cannot be accurately mapped to specific components, which is not conducive to rapid handling by maintenance personnel.

[0004] In summary, existing intelligent substation inspection technologies have significant shortcomings in terms of multimodal information fusion adaptability, comprehensiveness of abnormal feature capture, accuracy of identification and positioning, and dynamic adaptability of models. There is an urgent need for a technical solution that can integrate complementary multimodal information, adapt to dynamic operating conditions, accurately identify abnormal states and locations, and has continuous optimization capabilities, in order to improve the intelligence level and operational reliability of substation inspection. Summary of the Invention

[0005] This invention aims to address the problems in existing intelligent substation inspection technologies, such as incomplete coverage of single-modal data, inability of fixed weights to adapt to dynamic operating conditions, incomplete capture of abnormal features, insufficient positioning accuracy, and lack of adaptive update capabilities. It achieves this by simultaneously collecting multimodal time-series data and embedding contextual information, optimizing preprocessing and deep feature extraction processes, constructing a dynamic weighted fusion network with operating condition awareness, employing a spatiotemporal dual-stream architecture to jointly capture static and dynamic features of equipment, and combining multi-dimensional information to achieve accurate anomaly positioning and online model updates. This comprehensively improves the accuracy, robustness, and scenario adaptability of anomaly identification, reduces missed and false detections, simplifies operation and maintenance processes, and increases inspection efficiency, providing reliable intelligent technical support for the safe and stable operation of substation equipment.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] An anomaly identification method for substation intelligent inspection robots based on multimodal fusion and deep learning includes the following steps:

[0008] S1: The inspection robot synchronously collects multimodal time-series data of the target equipment in the substation, including visible light images, infrared images, audio signals and partial discharge signals, and embeds real-time GPS location information and the unique ID of the target equipment into the corresponding time-series data to complete the data contextual labeling;

[0009] Substation equipment exhibits diverse abnormalities (such as physical damage, abnormal temperature, abnormal mechanical noise, and insulation discharge). Single-modal data can only capture partial information (such as visible light only reflecting appearance and infrared only reflecting temperature). Multimodal data can achieve information complementarity and cover the characteristics of different types of abnormalities. The principle of embedding GPS location information with the unique ID of the equipment is to establish a correlation between data and physical scene, avoid confusion of data from multiple devices, provide a spatial index for subsequent abnormal location, and meet the needs of inspection scenarios where substation equipment is widely distributed and numerous.

[0010] S2: Preprocess and time-align the multimodal time series data. The preprocessing includes image enhancement, signal denoising and format conversion, and then the depth feature vectors of each modality are extracted.

[0011] Raw multimodal data contains problems such as environmental noise (e.g., light interference, electromagnetic noise), heterogeneous formats (images are two-dimensional matrices, signals are one-dimensional sequences), and asynchronous time (differences in acquisition frequencies of different sensors). Preprocessing can improve data quality and eliminate format barriers. The core of time alignment is to ensure the correlation between different modal data in the same time dimension, ensuring that the subsequent fused features can accurately reflect the state of the device at the same moment. The principle of deep feature extraction is to mine high-level abstract features of the data through neural networks, rather than relying on shallow features designed manually, thereby improving the identification of abnormal features.

[0012] S3: Input the depth feature vectors of each modality together with the real-time collected current working condition parameters into the dynamic weight fusion network. The dynamic weight fusion network outputs adaptive fusion weights for each modality that are adapted to the current working condition. The joint multimodal features are generated by weighting and fusing the depth feature vectors using the weights.

[0013] Substation operating conditions are dynamic (e.g., different ambient temperatures and equipment load rates), and the contribution of each mode to anomaly identification varies (e.g., the infrared image mode should have a higher weight for identifying overheating anomalies under high temperature conditions than other modes, and the partial discharge signal mode is more critical when the equipment is under high load). Fixed weights cannot adapt to complex operating conditions. The core principle of dynamic weight fusion network is to perceive the current environment and equipment operating status through operating condition parameters, dynamically adjust the weights of each mode, and make the joint features focus on the most discriminative information under the current operating conditions, thereby improving the effectiveness of fusion features.

[0014] S4: Based on the joint multimodal features, combined with the GPS location information embedded in S1 and the unique ID of the target device, a deep learning network is used to identify the abnormal state, type and abnormal location of the target device, and the identification result with confidence is output.

[0015] The joint multimodal feature integrates complementary information from various modalities. The deep learning network, through a spatiotemporal dual-stream architecture, simultaneously captures the spatial features (current static state, such as appearance damage and temperature distribution) and temporal features (dynamic trends, such as continuous temperature rise and periodic discharge signals) of the device. This solves the problems that single spatial features cannot identify gradual anomalies and single temporal features have weak anti-interference capabilities. The confidence output principle is to quantify the reliability of the identification results, providing decision-making reference for maintenance personnel and avoiding ineffective maintenance caused by misjudgment.

[0016] The preprocessing described in step S2 specifically includes:

[0017] The contrast-limited adaptive histogram equalization algorithm (CLAHE) was used for visible light images, with parameters set to a block size of 8×8 pixels and a contrast limiting factor of 2.0.

[0018] A temperature linear stretching algorithm is used on the infrared image to stretch the original temperature value. Mapped to the [0,255] grayscale space;

[0019] The original temperature range of infrared images varies depending on the type of equipment and the ambient temperature (e.g., the normal temperature of a transformer is 30-60℃, but it can reach over 100℃ in abnormal situations). Directly inputting these values ​​into the network will lead to feature extraction deviations due to the inconsistent numerical ranges. The core of linear temperature stretching is to normalize any temperature range to a standard grayscale space, so that infrared images in different scenarios have a unified numerical scale, while highlighting temperature difference features, which facilitates the subsequent identification of overheating anomalies.

[0020] The audio signal was denoised using wavelet thresholding with a db4 wavelet basis, 5 decomposition levels, and a soft thresholding function. ,in , The standard deviation of noise. The signal length;

[0021] Substation audio signals contain environmental noise (such as fan noise and conversations) and electromagnetic interference, and are non-stationary signals. Wavelet transform can decompose audio signals into wavelet coefficients of different scales, with signal energy concentrated in a few large coefficients and noise distributed in small coefficients. The db4 wavelet basis has good time-frequency localization characteristics, making it suitable for non-stationary signal processing. The soft threshold function smooths and shrinks the wavelet coefficients, avoiding signal abrupt changes caused by the hard threshold function. It removes noise while retaining the core characteristics of equipment noise (such as vibrations from mechanical loosening and friction sounds from components).

[0022] The partial discharge signal is filtered using a moving average filter with a window size of 50 sampling points.

[0023] Partial discharge signals are high-frequency pulse signals, which are easily affected by high-frequency noise from the power grid and sensor interference. Moving average filtering reduces the fluctuation amplitude of random noise by averaging 50 consecutive sampling points, while preserving the trend characteristics of the signal (such as the peak value and duration of the discharge signal), avoiding noise from masking the true discharge signal, and providing a clean signal source for the identification of partial discharge anomalies.

[0024] The preprocessed visible light / infrared images were uniformly converted into RGB three-channel tensors, the audio signals were converted into Mel-frequency spectrograms, and the partial discharge signals were converted into time-frequency graphs. All modal data were unified into time-series tensor format. , For timing length, For single-frame feature dimensions;

[0025] The original formats of data from different modalities are heterogeneous (images are two-dimensional matrices, audio is a one-dimensional sequence, and partial discharge is a pulse signal), making it impossible to directly input them into the same network for fusion processing. The core of format conversion is to map all data to a unified "time-feature" dimension. The time tensor format makes it easier for the network to capture the changing trends of each modal data in the time dimension (such as gradually increasing temperature and increasing discharge signal frequency), laying the foundation for subsequent time feature extraction.

[0026] The deep feature vector extraction in step S2 specifically includes:

[0027] Visible light image feature extraction uses a ResNet-50 network, removing the last fully connected layer while retaining the convolutional basis output, and the feature vector dimension... ;

[0028] Visible light images need to identify subtle anomalies in the appearance of the device (such as cracks in the casing or missing connectors), requiring the extraction of high-dimensional, fine-grained features. ResNet-50 solves the gradient vanishing problem in deep networks through residual connections, enabling the construction of deeper network structures and the extraction of richer abstract features. Removing the last fully connected layer is to preserve the spatial feature mapping of the convolutional layer output, avoiding the loss of local detail information by the fully connected layer. The 2048-dimensional feature vector can fully represent various state information of the device's appearance.

[0029] Infrared image feature extraction employs a lightweight MobileNetV3 network, with a feature vector dimension of [missing information]. ;

[0030] The core information of infrared images is temperature distribution, which does not require overly complex feature extraction, and the embedded processors on the inspection robots have limited computing resources. MobileNetV3 uses depthwise separable convolution and lightweight activation functions to significantly reduce the amount of computation and parameters while ensuring feature extraction capabilities, making it suitable for embedded computing environments. The 1280-dimensional feature vector can balance feature representation capabilities and computational efficiency, accurately capturing temperature difference features in infrared images.

[0031] Audio signal feature extraction uses a pre-trained VGGish network, and the output feature vector dimension is... ;

[0032] Abnormal noises from substation equipment (such as mechanical loosening or component failure) belong to specific audio scenarios, and directly training the network requires a large amount of labeled data. VGGish is a network pre-trained on a large-scale audio dataset, which has learned general audio features (such as frequency and amplitude changes). Through transfer learning, it can be quickly adapted to the substation abnormal noise recognition task, reducing the need for labeled data. The 128-dimensional feature vector is a low-dimensional and compact feature that can effectively represent the core characteristics of abnormal noises, while reducing the complexity of subsequent fusion calculations.

[0033] Partial discharge signal feature extraction employs a one-dimensional convolutional neural network (1D-CNN), with the output feature vector dimension... ;

[0034] Partial discharge signals are one-dimensional time-series pulse signals. 1D-CNN can directly process one-dimensional data without dimension transformation. It captures local features of the signal (such as pulse peak, width, and interval) by sliding convolution kernels. Compared with traditional manual feature extraction methods (such as statistical features), it can automatically mine nonlinear and deep discharge features. The 128-dimensional feature vector ensures feature recognition while avoiding feature redundancy and adapts to the computational needs of multimodal fusion.

[0035] The depth feature vectors of each modality are defined as follows: ,in For the first The feature vector of a single frame at a given time;

[0036] Temporal feature vectors can record the process of device state changes over time, rather than just capturing the state at a single moment. For example, device overheating is usually manifested as a gradual increase in temperature over time. Temporal features can reflect this gradual trend, helping the network distinguish between sudden noise and real anomalies, and improving the robustness of recognition.

[0037] The construction of the dynamic weight fusion network in step S3 includes a network design principle based on condition awareness to achieve adaptive adjustment of modal weights, thereby improving the relevance of the fused features.

[0038] The network input consists of deep feature vectors for each modality. and current operating condition sensing parameters ,in Includes environmental parameters: temperature ,humidity Equipment operating parameters: load rate ,Voltage ; A total of 4-dimensional vectors;

[0039] Operating parameters directly affect the recognition performance of each mode: when the ambient temperature is high, the temperature resolution of the infrared image is affected, and its weight needs to be reduced; when the equipment load rate is high, the risk of partial discharge increases, and the weight of the partial discharge signal mode needs to be increased; ambient humidity affects the insulation performance of the equipment, which is indirectly related to partial discharge and infrared temperature characteristics; voltage parameters reflect the operating load of the equipment and are closely related to overheating and abnormal discharge; by incorporating operating parameters into the input, the network can perceive the particularity of the current scene and provide a basis for dynamic weight calculation.

[0040] The network structure consists of two layers:

[0041] Feature concatenation layer: Concatenates the features of each modality into a joint input according to time steps. ;

[0042] By concatenating the components at each time step, we can ensure that the modal features and operating parameters at the same moment correspond one-to-one. This allows the network to dynamically allocate weights based on the current operating conditions and the quality of each modal feature when calculating weights. The joint input integrates all relevant information, avoids information fragmentation, and ensures the comprehensiveness of weight calculation.

[0043] Attention weight calculation layer: Employs a multi-head self-attention mechanism, with a certain number of heads. Calculate the attention score for each modality. : ,in For input The query, key, and value matrix after linear transformation It is a single-modal feature dimension;

[0044] The multi-head self-attention mechanism can capture the dependencies between different modalities and the temporal dependencies within a modality in parallel: the four attention heads focus on four dimensions, namely "modality-condition", "modality-modality", "modality-time" and "global information", which can more comprehensively evaluate the importance of each modality compared with single-head attention. Calculate the similarity between the query and the key to reflect the degree of matching between modal features and operating conditions and other modalities. This is used for normalization to avoid excessively high similarity values ​​that could saturate the Softmax function. Through this mechanism, the network can automatically identify the most discriminative mode under the current operating conditions.

[0045] Output adaptive fusion weights for each modality , And satisfy ;

[0046] The Sigmoid function maps attention scores to the [0,1] interval, ensuring the effectiveness of the weights; normalization makes the weight allocation conform to probabilistic logic, and the contribution of each modality can be quantified, avoiding feature redundancy caused by weight superposition; this design gives the weights a clear physical meaning, which is convenient for subsequent result interpretation.

[0047] The weighted fusion method described in step S3 generates joint multimodal features. The calculation formula is:

[0048]

[0049] in This is the time-averaged vector of single-modal features. The modality output by the dynamic weight fusion network The weights;

[0050] The principle behind calculating the time-averaged vector is that time-series data contains certain redundant information and random fluctuations. Averaging can extract the core stable features of the modes and reduce the impact of instantaneous noise (such as image blurring or signal interference at a certain moment). The core principle of weighted fusion is to allocate the contribution of each mode according to dynamic weights, so that the joint features focus on the key information under the current operating condition: for example, when the equipment overheats, the infrared mode weights... The increase in the proportion of time-averaged features in the joint features highlights temperature anomaly information; the weighted superposition of multimodal core features achieves information complementarity and avoids recognition errors caused by the failure of a single modality.

[0051] The deep learning network described in step S4 adopts a spatiotemporal dual-stream network architecture. The network design principle is to combine spatial static features and temporal dynamic features to improve the accuracy and robustness of anomaly detection, specifically including:

[0052] Spatial branching: using joint multimodal features Using this as input, spatial features are extracted through a two-layer fully connected network. ;

[0053] The joint multimodal features have integrated the core information of each modality. The role of the spatial branch is to explore the spatial correlation between features (such as the correlation between equipment appearance damage and local temperature rise). The two-layer fully connected network can map high-dimensional fused features to low-dimensional compact spatial feature vectors, while introducing nonlinear transformation to enhance the discriminative ability of features. The 256-dimensional feature vector balances feature expressive power and computational efficiency, making it suitable for subsequent classification tasks.

[0054] Time branching: using the original temporal features of each modality As input, time-dependent features are extracted using a bidirectional LSTM network. ;

[0055] Equipment anomalies often manifest as dynamic changes in state (such as continuous temperature rise or increased frequency of discharge signals). Unidirectional LSTM can only capture the time dependency from the past to the present, while bidirectional LSTM, through forward LSTM (capturing the trend from the past to the present) and backward LSTM (capturing the trend from the present to the future), comprehensively captures the sequential dependencies of time series data and accurately identifies gradual and periodic anomalies. The 256-dimensional time feature vector has the same dimension as the spatial feature vector, which facilitates subsequent fusion.

[0056] Fusion classification layer: and After concatenation, the input is fed into a fully connected layer, activated by Softmax, and the output is an anomaly probability distribution, including normal states. ,overheat Mechanical loosening Partial discharge Different sounds There are 5 categories in total;

[0057] Spatial features reflect the current static state of the equipment, while temporal features reflect the dynamic trend of the equipment's state. The combination of the two achieves a comprehensive "static + dynamic" feature representation, solving the problems that single static features cannot identify gradual anomalies and single dynamic features have weak anti-interference capabilities. The Softmax activation function converts the classification score into a probability distribution, making it easier to intuitively judge the probability of anomaly types. The five types of anomalies cover the most common fault types of substation equipment, meeting the actual inspection needs.

[0058] The network loss function uses weighted cross-entropy loss: ,in For the sample size, For the sample The true label, To predict probabilities, For category weights, , ;

[0059] The number of normal equipment samples in a substation is far greater than the number of abnormal samples, leading to class imbalance. Traditional cross-entropy loss will cause the model to favor predicting the majority class (normal samples); weighted cross-entropy loss reduces the weight of normal samples ( ), increase the weight of outlier samples ( This forces the model to focus on learning minority class anomalies, balances the training loss of each class, and improves the accuracy of anomaly identification.

[0060] The method for determining the abnormal location in step S4 is to achieve precise location of the abnormal location based on multi-dimensional information mapping. The principle is to combine image spatial features, signal temporal features, and physical coordinate information to transform abstract abnormal features into specific device component locations, specifically including:

[0061] Visible / infrared image features , Generate a heatmap , , Image height and width;

[0062] Image features , There is a spatial correspondence with the original image. The core of heatmap generation is to map the feature response back to the pixel space of the original image through backpropagation. The higher the feature response intensity (such as the appearance features of abnormal areas or high temperature features), the higher the gray value of the corresponding position in the heatmap. The peak coordinates of the heatmap directly correspond to the abnormal areas in the image, providing an intuitive basis for spatial positioning.

[0063] Characteristics of audio / partial discharge signals , Time-frequency analysis can be used to locate periods of concentrated abnormal energy. , ;

[0064] Abnormal sounds in audio signals and pulses in partial discharge signals both correspond to specific time-frequency energy distributions. Time-frequency analysis (such as short-time Fourier transform) can map one-dimensional signals to the time-frequency domain, identifying periods of concentrated abnormal energy. , The time of occurrence of the corresponding signal anomaly can be determined by combining the movement trajectory of the inspection robot (obtained through GPS location time-series data), thus providing a time-dimensional constraint for positioning.

[0065] Combined with GPS location information: Unique ID of the target device: Corresponding preset component coordinate set Map the spatial location corresponding to the peak coordinates or abnormal time period of the heat map to the equipment component and output the abnormal location;

[0066] GPS location information provides the device's global spatial coordinates. The target device's unique ID is associated with a preset 3D model of the device (including the coordinate set of each component), realizing a hierarchical mapping of "global coordinates → device → component". The peak coordinates of the heat map correspond to local areas in the device image. Through image calibration (establishing a mapping relationship between image pixel coordinates and physical coordinates of device components), the specific component (such as transformer terminals or insulators) can be accurately located.

[0067] It also includes online updates of the deep learning network, specifically including:

[0068] Define network performance evaluation metrics , Time period Number of samples within, For indicator functions, For predicting labels;

[0069] This indicator is for a time period. The accuracy rate within the time frame directly reflects the actual performance of the model. Accuracy was chosen as the evaluation metric because it is intuitive, easy to calculate, and comprehensively reflects the model's ability to identify various anomalies. The time frame is calculated using a sliding window. The average accuracy is improved to avoid misjudgments caused by fluctuations in samples at a single moment.

[0070] when When online updates are triggered, the baseline accuracy is updated. Allowed threshold to decrease ;

[0071] Benchmark accuracy The established performance standard for the model meets the high accuracy requirements of substation inspections; a lower threshold is allowed. This is to avoid unnecessary updates triggered by short-term sample anomalies (such as data quality degradation caused by extreme weather), and to balance model stability and adaptability; when the accuracy is below 90%, it means that the model can no longer adapt to the current scenario (such as new equipment or environmental changes), and needs to be updated in time.

[0072] The update function uses the Elastic Weight Integration (EWC) algorithm, and a regularization term is added to the loss function: ,in For cross-entropy loss, The regularization coefficient is . This is the parameter importance matrix. , The network is fine-tuned using newly collected labeled samples, with the current and old network parameters set to [value]. The learning rate is set to [value]. ;

[0073] The EWC algorithm protects important parameters of older networks through regularization. (The larger the value, the more important the parameter), to avoid losing old knowledge; To strengthen the regularization coefficient and ensure the stability of the old parameters; the newly collected labeled samples are the latest data of the current scene, containing newly added abnormal features, and fine-tuning the network enables the model to learn new knowledge; A low learning rate avoids drastic changes in parameters and ensures the stability of the model after updates.

[0074] The substation intelligent inspection robot anomaly recognition system based on multimodal fusion and deep learning includes a data acquisition module, a preprocessing and feature extraction module, a dynamic weight fusion module, and an anomaly recognition and localization module. These modules are sequentially connected and their specific functions are as follows:

[0075] Data acquisition module: Equipped with the inspection robot, it is used to simultaneously acquire visible light images, infrared images, audio signals, and partial discharge signals of target equipment in the substation, obtain real-time GPS location information, and embed the GPS location information and the unique ID of the target equipment into the corresponding time-series data to complete the data contextual labeling;

[0076] Preprocessing and feature extraction module: Receives multimodal time-series data output from the data acquisition module, performs preprocessing operations first, and then extracts the depth feature vectors of each modality through a preset network;

[0077] Dynamic weight fusion module: Receives the deep feature vectors of each modality output by the preprocessing and feature extraction module, and obtains the current working condition parameters. It outputs the adaptive fusion weights of each modality through the built-in dynamic weight fusion network. Based on these weights, it weights and fuses the time-averaged feature vectors of each modality to generate joint multimodal features.

[0078] Anomaly identification and localization module: It has a built-in spatiotemporal dual-stream deep learning network, which receives joint multimodal features, original temporal features of each modality, embedded GPS location information, and the unique ID of the target device. It extracts spatial features through a spatial branch fully connected network and extracts temporal dependent features through a temporal branch bidirectional LSTM network. After fusion classification layer, it outputs the abnormal state, abnormality type and corresponding confidence level of the target device. At the same time, it locates the specific location of the anomaly by generating image heatmaps, combining signal time and frequency analysis with GPS location and the device's preset component coordinate set.

[0079] It also includes a data storage module, a communication module, a visualization interaction module, and an online model update module, with the following specific functions:

[0080] Data storage module: It communicates with the data acquisition module and the anomaly recognition and localization module to store multimodal time-series data after scene-based labeling, preprocessing results, deep feature vectors, joint multimodal features, and anomaly recognition results and location information with confidence.

[0081] Communication module: It adopts wireless communication technology to realize data transmission between the data acquisition module, preprocessing and feature extraction module, dynamic weight fusion module, and anomaly identification and location module. At the same time, it uploads the anomaly identification results to the substation monitoring center in real time and receives control commands from the monitoring center.

[0082] Visualization and Interaction Module: Located in the monitoring center, it is used to visually display the multimodal data collected by the target device, abnormal status, abnormal type, abnormal location and confidence level, and supports historical data query and abnormal record tracing;

[0083] Online model update module: It communicates with the anomaly detection and localization module and the data storage module to calculate the performance index of the spatiotemporal dual-stream deep learning network in real time. When the index is lower than the baseline accuracy and the decrease exceeds 0.05, it triggers an online update.

[0084] The present invention has the following beneficial effects:

[0085] Multimodal information complementarity comprehensively covers anomaly types. This invention uses an inspection robot to simultaneously collect visible light images, infrared images, audio signals, and partial discharge signals, integrating multi-dimensional information such as equipment appearance, temperature, sound, and electrical signals. This overcomes the limitation of single-modal data only capturing local features, and can comprehensively cover common anomaly types in substation equipment such as overheating, mechanical loosening, partial discharge, and abnormal noise. It effectively avoids missed detections due to incomplete information and significantly improves the completeness of anomaly identification.

[0086] Dynamic weight fusion adapts to complex operating conditions and enhances feature discrimination. Relying on a dynamic weight fusion network, combined with real-time operating parameters such as ambient temperature and humidity, equipment load rate, and system voltage, the weights of each modality are adaptively adjusted through a multi-head self-attention mechanism. This allows the joint features to focus on the most discriminative information under the current operating conditions (e.g., strengthening the infrared modality weight under high temperature conditions and increasing the weight of partial discharge signals under high load). This solves the pain point that fixed weights cannot adapt to the dynamic operating conditions of substations, significantly enhances the relevance and effectiveness of fused features, and improves the robustness of anomaly identification in complex scenarios.

[0087] The spatiotemporal dual-stream architecture accurately captures features and improves recognition accuracy. The deep learning network adopts a dual-stream design with spatial and temporal branches. The spatial branch mines the spatial correlation of joint multimodal features, while the temporal branch captures the sequential dependencies of temporal data of each modality through bidirectional LSTM, realizing comprehensive feature representation and effectively identifying gradual anomalies (such as continuous temperature rise) and periodic anomalies (such as periodic occurrence of discharge signals). At the same time, a weighted cross-entropy loss is used to balance the differences between normal and abnormal samples, further improving the recognition accuracy of abnormal samples and reducing the probability of false detection.

[0088] Multi-dimensional positioning pinpoints specific components, facilitating efficient operation and maintenance. By combining image heatmap generation, signal time-frequency analysis, GPS location information, and pre-set component coordinate sets, abstract anomaly characteristics are mapped to specific equipment components (such as transformer terminals and insulators). This overcomes the limitations of traditional equipment-level positioning, providing maintenance personnel with precise guidance to anomaly locations, avoiding blind troubleshooting, significantly shortening fault handling time, and reducing operation and maintenance costs.

[0089] The online update mechanism ensures long-term adaptability and reduces maintenance costs. By monitoring the model's recognition accuracy in real time, when the performance falls below the baseline and the decline exceeds the threshold, the Elastic Weight Integration (EWC) algorithm is used to fine-tune the network parameters. This process preserves old knowledge while learning new anomaly features, allowing the model to adapt to new equipment in substations, changes in anomaly types, or long-term evolution of operating conditions without frequent retraining. This ensures long-term stable operation of the model and significantly reduces technical maintenance costs. Attached Figure Description

[0090] Figure 1This is a flowchart of the anomaly identification method for intelligent substation inspection robots based on multimodal fusion and deep learning proposed in this invention. Detailed Implementation

[0091] The present invention will be further described in detail below with reference to specific embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0092] I. Implementation Preparation:

[0093] (a) Hardware deployment

[0094] Inspection robot configuration: Equipped with a multimodal sensor suite, GPS positioning module, temperature and humidity sensor and equipment operation parameter acquisition unit. The core processor must support deep learning model inference to ensure synchronous acquisition and real-time processing of multimodal data.

[0095] Backend system deployment: Set up edge servers (for model training and online updates) and monitoring center servers (including data storage and application service nodes), and configure wireless communication equipment to realize data transmission between the robot and the monitoring center.

[0096] Equipment coordinate calibration: Assign a unique ID to the target equipment in the substation, preset the physical coordinate set of each equipment and component, establish the mapping relationship between GPS coordinates and equipment component coordinates, and provide a spatial index for anomaly location.

[0097] (II) Software Environment Configuration

[0098] Install PyTorch 1.12.1, configure data processing libraries such as OpenCV 4.5.5, NumPy, and SciPy, and deploy the dependencies related to ResNet-50, MobileNetV3, VGGish (pre-trained), one-dimensional CNN, and bidirectional LSTM networks.

[0099] Model preprocessing: Based on the multimodal dataset of substation equipment (including normal and abnormal samples such as overheating, mechanical loosening, partial discharge, and abnormal noise), the initial model of the spatiotemporal dual-stream deep learning network is trained, the loss function category weights are set, and the baseline accuracy is determined.

[0100] II. Implementation Steps of Anomaly Detection Method

[0101] Step 1: Multimodal time series data acquisition and contextual labeling

[0102] The inspection robot is activated and moves along the preset inspection path, simultaneously triggering various sensors to collect data: visible light images (capturing equipment appearance), infrared images (capturing temperature distribution), audio signals (capturing operating sounds), and partial discharge signals (capturing insulation discharge pulses). The acquisition frequency is adapted to the sensor performance to ensure the continuity of time-series data.

[0103] The latitude and longitude coordinates (x, y) output by the GPS positioning module are obtained in real time. The unique ID of the current inspection target device is extracted, and the GPS location information and device ID are embedded into each set of synchronously collected multimodal data to form contextualized labeled data with "data-location-device" association, avoiding data confusion among multiple devices.

[0104] Step 2: Data preprocessing, temporal alignment, and deep feature extraction

[0105] Preprocessing steps:

[0106] Visible light images: The Limiting Contrast Adaptive Histogram Equalization (CLAHE) algorithm is used, with a block size of 8×8 pixels and a contrast limiting factor of 2.0, to enhance image details.

[0107] Infrared images: A temperature linear stretching algorithm is used to map the original temperature value T_raw to the [0,255] grayscale space to unify the numerical scale.

[0108] Audio signal: Environmental noise was removed using a wavelet thresholding method with a db4 wavelet basis and 5-level decomposition.

[0109] Partial discharge signal: A moving average filter (window size 50 sampling points) is used to reduce high-frequency interference.

[0110] Format conversion and time alignment:

[0111] The preprocessed visible light / infrared images are converted into RGB three-channel tensors, the audio signals are converted into Mel spectrograms, and the partial discharge signals are converted into time-frequency diagrams. All modal data are unified into time-series tensor format.

[0112] Based on the timestamps collected by sensors, time-series data of different modalities are time-aligned to ensure that multimodal data at the same time correspond to the same device state.

[0113] Deep feature extraction:

[0114] Visible light image: Input ResNet-50 network (remove the last fully connected layer), output 2048-dimensional feature vector.

[0115] Infrared image: Input to MobileNetV3 network, output 1280-dimensional feature vector.

[0116] Audio signal: input to pre-trained VGGish network, output 128-dimensional feature vector.

[0117] Partial discharge signal: input to a one-dimensional CNN network, output a 128-dimensional feature vector.

[0118] Step 3: Dynamic weight fusion to generate joint multimodal features

[0119] Collect current operating parameters: ambient temperature and humidity (from temperature and humidity sensors), equipment load rate and system voltage (from equipment operating parameter acquisition unit) to form a 4-dimensional operating condition vector.

[0120] Constructing the joint input: The depth feature vectors of each modality are concatenated at time steps and incorporated into the operating parameters to obtain the joint input data.

[0121] Calculate the adaptive fusion weights:

[0122] Attention scores for each modality are calculated using a multi-head self-attention mechanism (h=4 heads). Modal weights are then obtained by mapping using the Sigmoid function.

[0123] Generate joint features: Calculate the time average vector of each modality to generate joint multimodal features.

[0124] Step 4: Anomaly Identification and Location

[0125] Abnormal Status and Type Identification:

[0126] Spatial branch: Input the joint multimodal features into a 2-layer fully connected network to extract 256-dimensional spatial features.

[0127] Temporal branch: Input the original temporal features of each modality into a bidirectional LSTM network to extract 256-dimensional time-dependent features.

[0128] Fusion classification: Spatial features and temporal dependent features are spliced ​​together, input into a fully connected layer and activated by Softmax, and the output is an anomaly probability distribution to determine the anomaly state and its corresponding confidence level.

[0129] Abnormal location location:

[0130] Image-based localization: Generate heatmaps from visible light / infrared image features and obtain the coordinates of heatmap peaks.

[0131] Signal-based localization: Time-frequency analysis of audio / partial discharge signal characteristics is performed to locate periods of concentrated abnormal energy, and the corresponding spatial area is determined by combining GPS time-series data.

[0132] Coordinate mapping: Based on GPS location information (x,y), device unique ID and preset component coordinate set, the peak coordinates of the heat map or the spatial area of ​​abnormal time period are mapped to specific device components, and the abnormal location is output.

[0133] III. Collaborative Implementation of System Modules

[0134] Data acquisition module: Execute step 1 to transmit the scene-tagged multimodal time series data to the preprocessing and feature extraction module.

[0135] Preprocessing and feature extraction module: Execute step 2 and output the depth feature vectors of each modality to the dynamic weight fusion module.

[0136] Dynamic weight fusion module: Execute step 3 and output the joint multimodal features to the anomaly recognition and localization module.

[0137] Anomaly identification and location module: Execute step 4 to upload the anomaly status, type, and location information with confidence level to the monitoring center through the communication module, and simultaneously store it in the data storage module.

[0138] Visualization and Interaction Module: Displays multimodal collected data, anomaly information, and confidence levels in the monitoring center, and supports historical data query and anomaly tracing.

[0139] IV. Implementation of Online Updates for Deep Learning Networks

[0140] Performance monitoring: The online model update module calculates the recognition accuracy of the deep learning network in real time within the time period t.

[0141] Update trigger: When the accuracy is too low or drops too quickly, an online update is triggered.

[0142] Model fine-tuning: The elastic weight integration algorithm is adopted to fine-tune the network parameters with a learning rate of 1e-5 using newly collected labeled samples, retaining old knowledge while learning new features.

[0143] V. Implementation and Verification

[0144] After each round of inspection, the anomaly identification results are compared with the manual inspection and annotation results to verify the accuracy, confidence level, and accuracy of anomaly location.

[0145] Regularly check the performance metrics of the deep learning network, confirm the effectiveness of the online update mechanism, and ensure that the deep learning network adapts to the dynamic operating conditions of the substation.

[0146] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An anomaly identification method for intelligent substation inspection robots based on multimodal fusion and deep learning, characterized in that: Includes the following steps: S1: The inspection robot synchronously collects multimodal time-series data of the target equipment in the substation, including visible light images, infrared images, audio signals and partial discharge signals, and embeds real-time GPS location information and the unique ID of the target equipment into the corresponding time-series data to complete the data contextual labeling; S2: Preprocess and time-align the multimodal time series data. The preprocessing includes image enhancement, signal denoising and format conversion, and then the depth feature vectors of each modality are extracted. S3: Input the depth feature vectors of each modality together with the real-time collected current working condition parameters into the dynamic weight fusion network. The dynamic weight fusion network outputs adaptive fusion weights for each modality that are adapted to the current working condition. The joint multimodal features are generated by weighting and fusing the depth feature vectors using the weights. S4: Based on the joint multimodal features, combined with the GPS location information embedded in S1 and the unique ID of the target device, the abnormal state, type and abnormal location of the target device are identified through a deep learning network, and the identification result with confidence is output.

2. The anomaly identification method for substation intelligent inspection robots based on multimodal fusion and deep learning according to claim 1, characterized in that: The preprocessing described in step S2 specifically includes: Image enhancement: For visible light images, a contrast-limited adaptive histogram equalization algorithm is used, with parameters set to a block size of 8×8 pixels and a contrast limit factor of 2.0; for infrared images, a temperature linear stretching algorithm is used to stretch the original temperature values. Mapped to the [0,255] grayscale space; Signal noise reduction: The audio signal is denoised using wavelet thresholding with a db4 wavelet basis, 5 decomposition levels, and a soft thresholding function. ,in , The standard deviation of noise. The signal length is given; the partial discharge signal is filtered using a moving average, with a window size of 50 sampling points. Format conversion: The preprocessed visible light / infrared images are uniformly converted into RGB three-channel tensors, audio signals are converted into Mel-frequency spectrograms, and partial discharge signals are converted into time-frequency graphs. All modal data are unified into time-series tensor format. , For timing length, The feature dimension is for a single frame.

3. The anomaly identification method for substation intelligent inspection robots based on multimodal fusion and deep learning according to claim 1, characterized in that: The deep feature vector extraction in step S2 specifically includes: Visible light image feature extraction uses a ResNet-50 network, removing the last fully connected layer while retaining the convolutional basis output, and the feature vector dimension... ; Infrared image feature extraction employs a lightweight MobileNetV3 network, with a feature vector dimension of [missing information]. ; Audio signal feature extraction uses a pre-trained VGGish network, and the output feature vector dimension is... ; Partial discharge signal feature extraction employs a one-dimensional convolutional neural network, with the output feature vector dimension... ; The depth feature vectors of each modality are defined as follows: ,in For the first The feature vector of a single frame at time t.

4. The anomaly identification method for substation intelligent inspection robots based on multimodal fusion and deep learning according to claim 1, characterized in that: The construction of the dynamic weight fusion network in step S3 includes: The network input consists of deep feature vectors for each modality. and current operating condition sensing parameters ,in Includes environmental parameters: temperature ,humidity Equipment operating parameters: load rate ,Voltage ; A total of 4-dimensional vectors; The network structure consists of two layers: Feature concatenation layer: Concatenates the features of each modality into a joint input according to time steps. ; Attention weight calculation layer: Employs a multi-head self-attention mechanism, with a certain number of heads. Calculate the attention score for each modality. : ,in For input The query, key, and value matrix after linear transformation It is a single-modal feature dimension; Output adaptive fusion weights for each modality , And satisfy .

5. The anomaly identification method for substation intelligent inspection robots based on multimodal fusion and deep learning according to claim 1, characterized in that: The weighted fusion method described in step S3 generates joint multimodal features. The calculation formula is: in This is the time-averaged vector of single-modal features. The modality output by the dynamic weight fusion network The weight.

6. The anomaly identification method for substation intelligent inspection robots based on multimodal fusion and deep learning according to claim 1, characterized in that: The deep learning network described in step S4 adopts a spatiotemporal dual-stream network architecture, including: Spatial branching: using joint multimodal features Using this as input, spatial features are extracted through a two-layer fully connected network. ; Time branching: using the original temporal features of each modality As input, time-dependent features are extracted using a bidirectional LSTM network. ; Fusion classification layer: and After concatenation, the input is fed into a fully connected layer, activated by Softmax, and the output is an anomaly probability distribution, including normal states. ,overheat Mechanical loosening Partial discharge Different sounds There are 5 categories in total; The network loss function uses weighted cross-entropy loss: in For the sample size, For the sample The true label, To predict probabilities, For category weights, , To balance the difference in the number of normal samples and abnormal samples.

7. The anomaly identification method for substation intelligent inspection robots based on multimodal fusion and deep learning according to claim 1, characterized in that: The method for determining the abnormal location in step S4 is as follows: Visible / infrared image features , Generate a heatmap , , Image height and width; Characteristics of audio / partial discharge signals , Time-frequency analysis can be used to locate periods of concentrated abnormal energy. , ; Combined with GPS location information: Unique ID of the target device: Corresponding preset component coordinate set It maps the peak coordinates of the heat map or the spatial location corresponding to the abnormal period to the equipment component and outputs the abnormal location.

8. The anomaly identification method for substation intelligent inspection robots based on multimodal fusion and deep learning according to claim 1, characterized in that: This also includes online updates to the deep learning network: Define network performance evaluation metrics , Time period Number of samples within, For indicator functions, For predicting labels; when When online updates are triggered, the baseline accuracy is updated. Allowed threshold to decrease ; The update function uses the elastic weight integration algorithm, and a regularization term is added to the loss function: in For cross-entropy loss, The regularization coefficient is . This is the parameter importance matrix. , The network is fine-tuned using newly collected labeled samples, with the current and old network parameters set to [value]. The learning rate is set to [value]. .

9. The substation intelligent inspection robot anomaly identification system based on multimodal fusion and deep learning according to claim 1, characterized in that: It includes a data acquisition module, a preprocessing and feature extraction module, a dynamic weight fusion module, and an anomaly detection and localization module. Each module is connected to the other in sequence, and their specific functions are as follows: Data acquisition module: Equipped with the inspection robot, it is used to simultaneously acquire visible light images, infrared images, audio signals, and partial discharge signals of target equipment in the substation, obtain real-time GPS location information, and embed the GPS location information and the unique ID of the target equipment into the corresponding time-series data to complete the data contextual labeling; Preprocessing and feature extraction module: Receives multimodal time-series data output from the data acquisition module, performs preprocessing operations first, and then extracts the depth feature vectors of each modality through a preset network; Dynamic weight fusion module: Receives the deep feature vectors of each modality output by the preprocessing and feature extraction module, and obtains the current working condition parameters. It outputs the adaptive fusion weights of each modality through the built-in dynamic weight fusion network. Based on these weights, it weights and fuses the time-averaged feature vectors of each modality to generate joint multimodal features. Anomaly identification and localization module: It has a built-in spatiotemporal dual-stream deep learning network, which receives joint multimodal features, original temporal features of each modality, embedded GPS location information, and the unique ID of the target device. It extracts spatial features through a spatial branch fully connected network and extracts temporal dependent features through a temporal branch bidirectional LSTM network. After fusion classification layer, it outputs the abnormal state, abnormality type and corresponding confidence level of the target device. At the same time, it locates the specific location of the anomaly by generating image heatmaps, combining signal time and frequency analysis with GPS location and the device's preset component coordinate set.

10. The substation intelligent inspection robot anomaly identification system based on multimodal fusion and deep learning according to claim 1, characterized in that: It also includes a data storage module, a communication module, a visualization interaction module, and an online model update module, with the following specific functions: Data storage module: It communicates with the data acquisition module and the anomaly recognition and localization module to store multimodal time-series data after scene-based labeling, preprocessing results, deep feature vectors, joint multimodal features, and anomaly recognition results and location information with confidence. Communication module: It adopts wireless communication technology to realize data transmission between the data acquisition module, preprocessing and feature extraction module, dynamic weight fusion module, and anomaly identification and location module. At the same time, it uploads the anomaly identification results to the substation monitoring center in real time and receives control commands from the monitoring center. Visualization and Interaction Module: Located in the monitoring center, it is used to visually display the multimodal data collected by the target device, abnormal status, abnormal type, abnormal location and confidence level, and supports historical data query and abnormal record tracing; Online model update module: It communicates with the anomaly detection and localization module and the data storage module to calculate the performance index of the spatiotemporal dual-stream deep learning network in real time. When the index is lower than the baseline accuracy and the decrease exceeds 0.05, it triggers an online update.

Images