Switch cabinet fault diagnosis method and system based on thermal imaging and voiceprint

Abstract

The invention discloses a switch cabinet fault diagnosis method and system based on thermal imaging and voiceprint. The method comprises the following steps: acquiring historical operation data, voiceprint data and thermal imaging data of a switch cabinet and an operation site picture of the switch cabinet; carrying out noise reduction processing on the acquired voiceprint data and historical operation data by adopting an infinite Kalman filter; constructing a space-time attention mixed expert model to carry out training prediction on the voiceprint data set and the historical operation data set, and obtaining predicted voiceprint data and historical operation data at the next moment; constructing an improved MaxViT-DF model to classify the collected thermal imaging data and the operation site picture, and training to obtain a fault feature image and a fault-free image; and fusing fault feature data output by the space-time attention hybrid expert model and the MaxViT-DF model by adopting multi-modal orthogonal sequence fusion. According to the invention, the accuracy and efficiency of fault diagnosis of the switch cabinet are improved, operation and maintenance personnel can find and process potential problems, and the reliability and safety of a power system are improved.

Description

Technical Field

[0001] The present invention relates to a method and system for fault diagnosis of switchgear, and particularly to a method and system for fault diagnosis of switchgear based on thermal imaging and acoustic fingerprint. Background Art

[0002] As an important part of the power system, the stable operation of switchgear is crucial for the reliability of the entire power system. Switchgear is a complete set of electrical equipment used in the power system, which plays roles such as switching, controlling, or protecting in power generation, transmission, distribution, power conversion, and consumption of the power system. It generally consists of a circuit breaker, current transformer, voltage transformer, lightning arrester, protection device, etc. Most of the on-line monitoring systems of switchgear can only simply collect and collate monitoring data, unable to provide reliable diagnostic results or maintenance suggestions, and unable to accurately diagnose the true health status of the equipment. However, traditional fault diagnosis methods usually rely on single-sensor data or simple threshold judgment, and this method often has poor effects when dealing with complex faults. Single-sensor data can only provide limited information and is difficult to comprehensively reflect the operating state of the equipment. Especially when facing complex faults with multi-factor coupling, it is easy to miss or misdiagnose. In addition, simple threshold judgment depends on fixed thresholds set by artificial experience and lacks adaptability to the dynamic characteristics of the equipment and environmental changes. When the operating state of the equipment is close to the threshold, it may not be able to accurately judge the severity of the fault, resulting in a decrease in the reliability of the diagnostic results. For example, in the fault diagnosis of switchgear, single-sensor data may not be able to effectively identify different types of faults such as partial discharge, temperature rise, and mechanical faults. Therefore, the limitations of traditional methods in complex fault diagnosis are gradually emerging, and it is difficult to meet the requirements of modern power systems for fault diagnosis accuracy and efficiency. Summary of the Invention

[0003] Object of the Invention: The object of the present invention is to provide a method and system for fault diagnosis of switchgear based on thermal imaging and acoustic fingerprint, improve diagnostic accuracy, ensure the normal operation of switchgear, and extend the equipment life.

[0004] Technical Solution: A method for fault diagnosis of switchgear based on thermal imaging and acoustic fingerprint according to the present invention includes the following steps:

[0005] (1) Collect historical operation data, acoustic fingerprint data, thermal imaging data, and on-site pictures of the switchgear operation;

[0006] (2) Use the unscented Kalman filter to perform noise reduction processing on the collected acoustic fingerprint data and historical operation data;

[0007] (3) Construct a time-enhanced spatio-temporal attention hybrid expert model to train and predict the acoustic fingerprint data set and historical operation data set, and obtain the predicted acoustic fingerprint data and historical operation data at the next moment;

[0008] (4) Introduce a deformable convolution module and a feature fusion module, and construct an improved MaxViT-DF model to train and classify the thermal imaging data and on-site operation pictures collected in step (1). After training, fault feature images and fault-free images are obtained.

[0009] (5) Adopt multi-modal orthogonal sequence fusion to fuse the data output in step (3) and the data with fault features output in step (4), and output a multi-modal representation after deep fusion to capture and express information from different modal images and voiceprints, and perform diagnostic analysis to complete fault diagnosis.

[0010] Preferably, the historical operation data in step (1) includes voltage, current, temperature and humidity, and is collected by voltage and current transformers and temperature and humidity sensors; the voiceprint data is collected by a voiceprint sensor; the thermal imaging data is collected by an infrared thermal imager; the on-site operation pictures of the switch cabinet are collected by a monitoring camera.

[0011] Preferably, the noise reduction processing in step (2) is specifically as follows:

[0012] (21) Initialization: Define state variables and an initial covariance matrix, where the state variables include voiceprint features and historical operation data, and set the state estimate value Covariance matrix P 0|0 ;

[0013] (22) Generate Sigma points: Generate Sigma points χ i using the state variable x and the covariance matrix P, where i = 0, 1,..., 2L;

[0014] (23) State prediction:

[0015] Predicted state

[0016]

[0017] Predicted covariance P k|k-1 :

[0018]

[0019] where Q is the process noise covariance matrix;

[0020] (24) Observation prediction:

[0021] Predicted observation

[0022]

[0023] Predicted Observation Covariance S k|k-1 :

[0024]

[0025] where h is the observation model, R is the observation noise covariance matrix, and W i m is the weight;

[0026] (25) Calculate the Kalman gain:

[0027]

[0028] where H k is the Jacobian matrix of the observation model.

[0029] (26) State update:

[0030] Update the state estimate

[0031]

[0032] where z k is the actual observation value;

[0033] Update the covariance matrix P k|k :

[0034] P k|k =(I - K k H k )P k|k-1

[0035] (27) Repeatedly iterate to update the state estimate and covariance matrix, and process each frame of voiceprint data and historical operation data to reduce the influence of noise.

[0036] Preferably, the spatio-temporal attention mixture-of-experts model described in step (3) includes 3 expert models, specifically:

[0037] Identity expert: Uses time information and does not consider the spatial relationship between switchgear sensors;

[0038] Science department static graph expert: Uses a learnable static graph to model the relationship between switchgear sensors;

[0039] Dynamic graph expert: Uses a dynamic graph to model the time-varying relationship between switchgear sensors;

[0040] During the prediction process, the spatial modeling methods used include: using an identity matrix, without considering the spatial relationship between switch cabinet sensors, and each sensor is processed independently; using a trainable adjacency matrix to construct the adjacency matrix by learning the hidden spatial information in the data to adapt to the dynamic changes of the sensors; using an attention mechanism to dynamically adjust the attention weights between each sensor.

[0041] Preferably, for the prediction of switch cabinet acoustic fingerprints and historical operation data using the spatio-temporal attention mixture expert model in step (3), the spatial attention is calculated using multi-head attention as follows:

[0042]

[0043]

[0044] Wherein, represents the final representation of node i; W O represents the weight matrix; represents the output of k heads; H s is the set of neighbor nodes; α i,j represents the attention weight of node i to its neighbor node j; e i,j represents the attention score; H i represents the set of neighbor nodes of node i; represents the value projection function of the k-th head; is the scaling factor.

[0045] Preferably, the deformable convolution module in step (4) is implemented as follows:

[0046] During sampling of the deformable convolution, for the input feature map:

[0047]

[0048] where H is the height, W is the width, and C is the number of channels;

[0049] The deformable convolution adds an offset Δp at the sampling position, and the output F dconv of the deformable convolution is expressed as:

[0050]

[0051] where P is the set of positions of the convolution kernel, Δp x and Δp y are the offsets in the x and y directions, and K(p) is the weight of the convolution kernel.

[0052] Preferably, the feature fusion module in step (4) inputs the input feature map into the MaxViT module and the CBR module respectively, wherein the CBR module is composed of convolution, batch normalization and ReLU activation functions.

[0053] Preferably, the improved MaxViT-DF model implementation process in step (4) is as follows:

[0054] (41) The deformable convolution and feature fusion modules are integrated into the original MaxViT model. The two convolutional layers in the original MaxViT are replaced by the deformable convolution for feature extraction. The deformable convolution extracts the input high-resolution image into a low-resolution feature map, which is close to the shape and position of the object to be identified when extracting features.

[0055] (42) The feature map after feature extraction is placed in the MaxViT block to apply attention, and the feature map is placed in the feature fusion module to fuse the global information. In each feature fusion process, the convolution layer is used to downsample the data to ensure that the size of the sampled data matches the size of the data processed by the MaxViT block;

[0056] (43) The results processed by the four MaxViT blocks are added to the results processed by the feature fusion module, and the classification results are output after being processed by the pooling layer and the fully connected layer.

[0057] Preferably, the multimodal orthogonal sequence fusion method in step (5) merges data in sequence step by step, and selectively weights the thermal imaging data with fault characteristics, the switch cabinet operation site pictures and voiceprint data, and the historical operation data; trains the single modal model, ranks the modes according to their respective performance, starts the fusion from the worst performing mode, and gradually moves to the best performing mode; uses the orthogonal loss function to improve the fusion performance and interpretability of the model;

[0058] The orthogonal loss function is expressed as:

[0059]

[0060] Among them, L is the loss function of a given task T, preds and outputs are the predictions and true values ​​of the model, respectively, and λ 1,i and λ 2,i are hyperparameters for the strength of the L1 and L2 regularization terms. The first term forces the absolute value of the dot product of the embedding vector to be close to zero, and the second term encourages the square of the dot product to be close to zero.

[0061] Selective weighting is performed by mapping into the weight vector:

[0062]

[0063] Among them, ai is the weight of the i-th mode, v i is the eigenvector of the i-th mode;

[0064] The method of the diagnostic analysis is as follows: identify the DC bias fault through the voiceprint recognition model. When a discharge fault occurs, the voiceprint recognition technology captures these signals; judge whether it is within the threshold range by comparing the historical operation data for fault diagnosis; detect the characteristic information of the thermal imaging and on-site images to diagnose whether there is a fault and determine the fault location; if there is a fault, start the alarm module to remind the staff to perform maintenance.

[0065] A switchgear fault diagnosis system based on thermal imaging and voiceprint according to the present invention includes:

[0066] Data acquisition module: used to acquire historical operation data, voiceprint data, thermal imaging data, and on-site images of the switchgear operation;

[0067] Data processing module: used to perform noise reduction processing on the acquired voiceprint data and historical operation data;

[0068] Model training module: used to predict the voiceprint set and historical operation data set by using the spatio-temporal attention hybrid expert model with time augmentation, and construct an improved MaxViT-DF model to train the thermal imaging data and on-site images of the switchgear operation;

[0069] Diagnosis module: used to capture the fault signal through the voiceprint recognition technology when a discharge fault occurs, judge whether it is within the threshold range by using the historical operation data, and diagnose whether there is a fault and determine the fault location by detecting the characteristic information of the thermal imaging and on-site images;

[0070] Alarm module: used to start the alarm module to remind the staff to perform maintenance when there is a fault;

[0071] Beneficial effects: Compared with the prior art, the present invention has the following remarkable advantages: (1) By adopting voltage and current transformers, temperature and humidity sensors, voiceprint sensors, infrared thermal imagers, and monitoring cameras, comprehensive acquisition of multi-dimensional data of the power system is realized, which helps to more accurately understand and analyze the operating state of the power system; (2) Using the unscented Kalman filter to denoise the voiceprint data and historical operation data significantly improves the accuracy and reliability of the data, making subsequent data analysis and processing more precise and providing higher-quality input data for fault diagnosis; (3) By constructing the TESTAM model to train and process the voiceprint set and historical operation data set, the recognition degree of the voiceprint data is improved, enabling the system to more accurately identify and distinguish different voiceprint features; (4) Constructing an improved MaxViT-DF model, introducing deformable convolution and feature fusion modules, effectively extracts image information, improves the accuracy of image processing, and enables the model to better understand and process image data; (5) Adopting the multi-modal orthogonal sequence fusion technology to deeply fuse the output data of the TESTAM model and the MaxViT-DF model, accurately captures and expresses information from different modalities, helps to improve the accuracy and efficiency of fault diagnosis, and enables the system to more quickly identify and respond to potential faults. Brief Description of the Drawings

[0072] Figure 1 It is a flow chart of the method described in the present invention.

[0073] Figure 2 It is a schematic diagram of the switch cabinet of the present invention.

[0074] Figure 3 It is a structural diagram of the improved MaxViT-DF model described in the present invention.

[0075] Figure 4 It is a structural diagram of the multi-modal orthogonal sequence fusion described in the present invention. Detailed Embodiments

[0076] The technical solution of the present invention will be further described below with reference to the drawings.

[0077] As Figure 1 shown, a switch cabinet fault diagnosis method based on thermal imaging and voiceprint is as follows:

[0078] (1) Collect historical operation data, voiceprint data, thermal imaging data, and on-site pictures of the switch cabinet operation.

[0079] As Figure 2As shown, the historical operation data includes voltage, current, temperature, and humidity, which are collected using voltage-current transformers and temperature-humidity sensors; the voiceprint data is collected using a voiceprint sensor; the thermal imaging data is collected using an infrared thermal imager; and the pictures of the switchgear operation site are collected using a surveillance camera.

[0080] (2) Use the unscented Kalman filter to perform noise reduction processing on the collected voiceprint data and historical operation data.

[0081] The implementation process of the noise reduction processing is as follows:

[0082] (21) Initialization: Define the state variables and the initial covariance matrix. The state variables include voiceprint features and historical data information, and the state estimation Set the covariance matrix P 0|0 ;

[0083] (22) Generate Sigma points: Use the state variables x and the covariance matrix P to generate Sigma points χ i , where i = 0, 1,..., 2L;

[0084] Weights W i 0 and W i m for the prediction and update steps;

[0085] (23) State prediction:

[0086] Predict the state

[0087]

[0088] Predict the covariance P k|k-1 :

[0089]

[0090] where Q is the process noise covariance matrix;

[0091] (24) Observation prediction:

[0092] Predict the observation

[0093]

[0094] Predict the observation covariance S k|k-1 :

[0095]

[0096] where h is the observation model and R is the observation noise covariance matrix;

[0097] (25) Calculate the Kalman gain:

[0098]

[0099] where H k is the Jacobian matrix of the observation model;

[0100] (26) State update:

[0101] Update the state estimate

[0102]

[0103] where z k is the actual observed value;

[0104] Update the covariance matrix P k|k :

[0105] P k|k = (I - K k H k ) P k|k-1

[0106] (27) Repeatedly iterate to update the state estimate value and the covariance matrix, and process each frame of voiceprint data and historical operation data to reduce the influence of noise.

[0107] (3) Construct a time-enhanced spatio-temporal attention mixture expert model to train and predict the voiceprint dataset and the historical operation dataset, and obtain the predicted voiceprint data and historical operation data for the next moment.

[0108] Input the data after noise reduction processing, including voiceprint, voltage, current, temperature, humidity, and air pressure, into the TESTAM model for training. The TESTAM model is a time-enhanced spatial attention model that combines the concept of mixture experts and predicts the future operation state of the switchgear through the input past data.

[0109] The TESTAM model consists of 3 expert models. Each expert has a different spatial modeling method, which are respectively:

[0110] Identity expert: Uses time information and does not consider the spatial relationship between switchgear sensors;

[0111] Science department static graph expert: Uses a learnable static graph to model the relationship between switchgear sensors;

[0112] Dynamic graph expert: Uses a dynamic graph to model the time-varying relationship between switchgear sensors;

[0113] Define the switchgear network:

[0114] G = (V, ε, A)

[0115] where V is the set of all sensors in the switchgear, |V| = N; ε represents a set of edges indicating the connectivity between sensors; A ∈ R N×N is the topological matrix of the graph;

[0116] The problem is formulated as a special multivariate time series prediction problem, predicting the future T graph signals based on the T' historical input graph signals. Therefore, a mapping function is established:

[0117]

[0118] Aiming to train a mapping function

[0119] f(·): R T′×N×C → R T×N×C

[0120] where N is the number of sensors and C is the number of input features.

[0121] In the prediction of the acoustic fingerprint and historical operation data of the switchgear, the following several spatial modeling methods are used:

[0122] (31) Using the identity matrix, that is, without spatial modeling, not considering the spatial relationship between the switchgear sensors, which is equivalent to using the identity matrix and processing each sensor independently.

[0123] (32) Using a trainable adjacency matrix, constructing the adjacency matrix by learning the hidden spatial information in the data, rather than relying on fixed prior knowledge, and being able to adapt to the dynamic changes of the sensors.

[0124] (33) Using the attention mechanism, which can dynamically adjust the attention weights between each sensor, so as to better capture the complex dynamic spatial dependence relationships, without the need for a pre-fixed connection relationship, but automatically adjusting by learning the patterns of the data.

[0125] The graph topology A is constructed by empirical rules, including inverse distance and cosine similarity. However, these empirically constructed graph structures are not necessarily optimal, so it will lead to poor spatial modeling quality.

[0126] Therefore, a trainable function g(·, θ) is used to derive the optimal topological representation as:

[0127]

[0128] The above spatial modeling methods can be divided into two subcategories according to whether g(·, θ) depends on X (t) as follows:

[0129] One is to define g(·, θ) = E ∈ RN×e , that is, using the identity matrix, which is independent of time and insensitive to noise, but there is less on-site modeling.

[0130] The second is the modeling of time-varying graph structure, using a trainable adjacency matrix, g(H (t) , θ) = H (t) W, W ∈ R d×e , projects the hidden state into another embedding space. Ideally, this method can model the dynamic changes of the graph topology, but it is sensitive to noise.

[0131] To reduce noise sensitivity and obtain a time-varying graph structure, a spatial attention mechanism is used for the prediction of switchgear acoustic fingerprints and historical operation data. Given node i and its spatial neighbors N i of the input H i , the spatial attention is calculated using multi-head attention as follows:

[0132]

[0133] where, represents the final representation of node i; W O represents the weight matrix; represents the output of k heads; H s is the set of neighbor nodes; α i,j represents the attention weight of node i to its neighbor node j; e i,j represents the attention score; H i represents the set of neighbor nodes of node i; represents the value projection function of the k-th head; is the scaling factor.

[0134] Since the time feature can be regarded as a global position with specific periodicity, the position embedding in the Transformer architecture is omitted in the model. Instead of normalizing the time feature, Time2Vec embedding is used for periodic and linear modeling.

[0135]

[0136] where, TIM(τ)[i] represents the i-th dimensional embedding vector of the time feature τ after passing through Time2Vec embedding; ω i represents the periodic scaling factor of the i-th dimensional embedding vector; υ(τ)[i] represents the periodic function value of the time feature τ in the i-th dimension, usually a periodic activation function such as a sine or cosine function. φ i represents the offset or phase adjustment of the i-th dimensional embedding vector. F represents a non-linear activation function; h represents the total number of dimensions of the embedded vector.

[0137] The temporal attention in TESTAM is the same as that in the Transformer. Attention is an attractive solution for temporal modeling because, unlike recurrent unit-based or convolutional-based temporal modeling, it can be used to directly focus on features across time steps without any restrictions. Temporal attention allows for parallel computation, which is beneficial for long-term sequence modeling. Additionally, TESTAM has fewer inductive biases in terms of locality and sequentiality. While strong inductive biases can help with training, fewer inductive biases can achieve better generalization.

[0138] The TESTAM model exhibits superior performance in predicting abnormal fluctuations in electrical switchgear. By predicting the acoustic fingerprint data and historical operation data in the next time period through the TESTAM model, faults can be detected more promptly, enabling effective fault diagnosis.

[0139] (4) Introduce a deformable convolution module and a feature fusion module to construct an improved MaxViT-DF model for training and classifying the thermal imaging data and on-site operation pictures collected in step (1). After training, fault feature images and fault-free images are obtained.

[0140] The deformable convolution can be closer to the shape and size of the object during sampling and has stronger robustness for the input feature map

[0141]

[0142] where H is the height, W is the width, and C is the number of channels.

[0143] The deformable convolution adds an offset Δp at the sampling position, enabling the model to be closer to the recognition target in a complex environment when extracting features. The output F of the deformable convolution dconv can be expressed as:

[0144]

[0145] where P is the set of positions of the convolution kernel, Δp x and Δp y are the offsets in the x and y directions, and K(p) is the weight of the convolution kernel.

[0146] The feature fusion module divides the input feature map into two parts, which respectively enter the MaxViT module to apply attention and the CBR module to extract features. The CBR module consists of a convolution, batch normalization, and ReLU activation function.

[0147] Let F maxvit be the output of the MaxViT module, F cbr be the output of the CBR module, and the output F of the feature fusion module fuse can be expressed as:

[0148] F fuse = F maxvit + F cbr

[0149] As Figure 3 shown, the implementation process of the MaxViT-DF model is as follows:

[0150] (41) Integrate the deformable convolution and feature fusion module into the original MaxViT model. Replace 2 convolutional layers in the original MaxViT with deformable convolution for feature extraction. The deformable convolution extracts the input high-resolution image into a low-resolution feature map, approaching the shape and position of the object to be identified when extracting features;

[0151] (42) Apply attention to the feature map after feature extraction in the MaxViT block, and put the feature map into the feature fusion module to fuse global information. In each feature fusion process, use the convolutional layer to downsample the data to ensure that the size of the sampled data matches the size of the data processed by the MaxViT block;

[0152] (43) Add the results processed by the MaxViT block and the results processed by the feature fusion module, and output the classification results after passing through the pooling layer and the fully connected layer.

[0153] Among them, the MaxViT block consists of MBConv and the multi-axis attention module. In MBConv, the input features are depthwise separable convolved after being upsampled by convolution. The SE module is used to enhance the model's attention ability to important channels, and then pointwise convolution is used again to restore the dimension; Since it is computationally infeasible to apply attention directly across the entire space, the multi-axis attention module is used to apply attention. The multi-axis attention module includes window attention and grid attention, thus forming two forms of local and sparse attention.

[0154] The window attention for the input feature map is transformed into a shape tensor

[0155]

[0156] denotes being divided into non-overlapping windows, where the size of each window is P×P. Finally, self-attention calculation is performed in each window as follows:

[0157]

[0158] Different from the traditional operation of dividing the feature map using a fixed window size, the grid attention uses a fixed G×G, and the uniform grid grids the input tensor into to obtain an adaptive window Finally, self-attention calculation is used on G×G. By using the same window P×P and grid G×G, the calculation between local and global can be effectively balanced, and it only has linear complexity.

[0159]

[0160] Connect between window-pane attention and grid attention through a feed-forward network.

[0161] Optimize using the AdamW optimizer with adaptive gradient descent method, with an initial learning rate of 0.0001 and a maximum number of training steps of 200. Use the BCEWithLogitsLoss loss function. First, normalize the predicted classes through the Sigmoid function, and then calculate the loss using the true labels and the probabilities of the predicted classes.

[0162] The calculation formula of the BCEWithLogitsLoss loss function is:

[0163]

[0164] Among them, is the Sigmoid function, t[i] is the true label of the i-th class, and o[i] is the probability of the i-th class prediction.

[0165] (5) Adopt multi-modal orthogonal sequence fusion to fuse the data output in step (3) and the data with fault features output in step (4), output the multi-modal representation after deep fusion, capture and express the information from different modalities of images and voiceprints, and conduct diagnostic analysis to complete fault diagnosis.

[0166] As Figure 4 shown, multi-modal orthogonal sequence fusion (OSF) is a method for fusing data from different sources in multi-modal learning.

[0167] The multi-modal orthogonal sequence fusion method gradually combines data in sequence. Different from the traditional method of fusing all data at once, multi-modal orthogonal sequence fusion allows selective weighting of different types of data.

[0168] Determine the best order of sequential fusion. If testing all possible combinations to sort the modalities, the computational cost is high and it is difficult to scale. Therefore, a ranking-based method is used to determine the fusion order. Train single-modal models, rank the modalities according to their respective performances, start fusing from the worst-performing modality, and gradually move towards the best-performing modality. Extract complementary information from each modality and fuse them in a way that improves the overall performance.

[0169] To encourage the modalities to maintain complementary features during the fusion process, the OSF uses an orthogonal loss function to improve the fusion performance and interpretability of the model. The loss function can be expressed as:

[0170]

[0171] where L is the loss function for a given task T, preds and outputs are the predictions and ground truths of the model respectively, and λ 1,i and λ 2,i are hyperparameters for the strengths of the L1 and L2 regularization terms. The first term forces the absolute value of the dot product of the embedding vectors to be close to zero, while the second term encourages the square of the dot product to be close to zero, jointly ensuring that the embedding vectors are not only diverse but also complementary.

[0172] The multi-modal orthogonal sequence fusion loss function can be optimized using the standard backpropagation algorithm. During training, the network learns to weight the modalities in a way that maximizes the diversity and complementarity of the fused embeddings at each layer, resulting in a more effective and robust fusion process.

[0173] Multi-modal orthogonal sequence fusion allows for selective weighting of the modalities, achieved by introducing a weight vector:

[0174]

[0175] where a i is the weight of the i-th modality, and v i is the feature vector of the i-th modality.

[0176] By fusing different modality data such as images, voiceprints, and historical operation data through multi-modal orthogonal sequence fusion, it shows excellent performance on various tasks. Compared with existing fusion technologies, it provides higher accuracy and can reveal the complementary relationships between modalities.

[0177] Selectively weight different types of data, extract the unique information of each data type, and send the fused result to the switchgear equipment management system for fault judgment.

[0178] The DC bias fault can be identified through the voiceprint recognition model. When a discharge fault occurs, these signals are captured through the voiceprint recognition technology; the remaining historical operation data is judged by comparison to determine whether it is within the threshold range for diagnosis; by detecting the characteristic information of the thermal imaging and on-site images, it is diagnosed whether there is a fault and the fault location is determined. If a fault exists, the alarm module is activated to remind the staff for maintenance.

[0179] The present invention provides a switchgear fault diagnosis system based on thermal imaging and voiceprint, including:

[0180] Data acquisition module: used to acquire historical operation data, voiceprint data, thermal imaging data, and on-site images of switchgear operation;

[0181] Data processing module: used to perform noise reduction processing on the acquired voiceprint data and historical operation data;

[0182] Model training module: used to predict the voiceprint set and historical operation data set using a spatio-temporal attention mixture expert model with temporal augmentation, and construct an improved MaxViT-DF model to train the thermal imaging data and on-site images of switchgear operation;

[0183] Diagnosis module: used to capture fault signals through voiceprint recognition technology when a discharge fault occurs, use historical operation data to determine whether it is within the threshold range, and diagnose whether there is a fault and determine the fault location by detecting the characteristic information of thermal imaging and on-site images;

[0184] Alarm module: used to start the alarm module to remind the staff to perform maintenance when there is a fault.

Claims

1. A switch cabinet fault diagnosis method based on thermal imaging and voiceprint, characterized in that: The following steps are involved: (1) Collect switchgear historical operation data, voiceprint data, thermal imaging data and switchgear operation site pictures; (2) Using the infinite Kalman filter to reduce noise on the collected voiceprint data and historical operation data; (3) Construct a time-enhanced spatiotemporal attention hybrid expert model to train and predict the voiceprint dataset and historical operation dataset to obtain the predicted voiceprint data and historical operation data at the next moment; (4) Introducing the deformable convolution module and the feature fusion module, constructing the improved MaxViT-DF model to train and classify the thermal imaging data and the operation site pictures collected in step (1), and obtaining images with fault features and images without faults after training; (5) Multimodal orthogonal sequence fusion is used to fuse the data output from step (3) and the data with fault characteristics output from step (4), and a multimodal representation after deep fusion is output to capture and express information from different modal images and voiceprints, and to perform diagnostic analysis to complete fault diagnosis.

2. A switch cabinet fault diagnosis method based on thermal imaging and voiceprint according to claim 1, characterized in that: The historical operation data in step (1) includes voltage, current, temperature and humidity, which are collected using voltage and current transformers and temperature and humidity sensors; the voiceprint data is collected using a voiceprint sensor; and the thermal imaging data is collected using an infrared thermal imager; The switch cabinet operation site pictures are collected by using a surveillance camera.

3. The switch cabinet fault diagnosis method based on thermal imaging and voiceprint according to claim 1 is characterized in that: The noise reduction process in step (2) is specifically as follows: (21) Initialization: Define state variables and initial covariance matrix, where state variables include voiceprint features and historical operation data, and set state estimation values Covariance matrix P 0|0 ; (22) Generate Sigma point: Generate Sigma point χ using state variable x and covariance matrix P i , where i = 0, 1, ..., 2L; (23) Status prediction: Prediction Status Prediction covariance P k|k-1 : Where Q is the process noise covariance matrix; (24) Observation and prediction: Predicted Observations Prediction observation covariance S k|k-1 : Where h is the observation model, R is the observation noise covariance matrix, and W i m is the weight; (25) Calculate the Kalman gain: Among them, H k is the Jacobian matrix of the observation model. (26) Status update: Update state estimate Among them, z k is the actual observed value; Update the covariance matrix P k|k : P k|k =(I-K k H k )P k|k-1 (27) Repeat the iteration to update the state estimation value and covariance matrix, process each frame of voiceprint data and historical operation data, and reduce the impact of noise.

4. The switch cabinet fault diagnosis method based on thermal imaging and voiceprint according to claim 1 is characterized in that: The spatiotemporal attention hybrid expert model in step (3) includes three expert models, specifically: Identity Expert: Uses temporal information and does not consider the spatial relationship between switchgear sensors; Science Department Static Graph Expert: Using learnable static graphs to model the relationships between switchgear sensors; Dynamic Graph Expert: Use dynamic graphs to model the time-varying relationships between switchgear sensors; In the prediction process, the spatial modeling method includes: using a unit matrix, ignoring the spatial relationship between the switch cabinet sensors, and processing each sensor independently; using a trainable adjacency matrix, constructing the adjacency matrix by learning the hidden spatial information in the data, and adapting to the dynamic changes of the sensors; using an attention mechanism, dynamically adjusting the attention weights between each sensor.

5. The switch cabinet fault diagnosis method based on thermal imaging and voiceprint according to claim 1 is characterized in that: In step (3), the spatiotemporal attention hybrid expert model is used to predict the switch cabinet voiceprint and historical operation data, and multi-head attention is used to calculate the spatial attention, as shown below: in, represents the final representation of node i; W O represents the weight matrix; represents the output of k heads; H s is the set of neighbor nodes; α i,j represents the attention weight of node i to its neighbor node j; e i,j represents the attention score; H i Represents the set of neighbor nodes of node i; represents the value projection function of the k-th head; is the scaling factor.

6. The switch cabinet fault diagnosis method based on thermal imaging and voiceprint according to claim 1 is characterized in that: The deformable convolution module in step (4) is implemented as follows: When deformable convolution is sampling, for the input feature map: Among them, H is the height, W is the width, and C is the number of channels; The deformable convolution adds an offset Δp to the sampling position, and the output F of the deformable convolution is dconv It is expressed as: Among them, P is the location set of the convolution kernel, Δp x and Δp y is the offset in the x and y directions, and K(p) is the weight of the convolution kernel.

7. The switch cabinet fault diagnosis method based on thermal imaging and voiceprint according to claim 1 is characterized in that: The feature fusion module in step (4) inputs the input feature map into the MaxViT module and the CBR module respectively, where the CBR module consists of convolution, batch normalization and ReLU activation functions.

8. The switch cabinet fault diagnosis method based on thermal imaging and voiceprint according to claim 1 is characterized in that: The improved MaxViT-DF model implementation process in step (4) is as follows: (41) The deformable convolution and feature fusion modules are integrated into the original MaxViT model. The two convolutional layers in the original MaxViT are replaced by the deformable convolution for feature extraction. The deformable convolution extracts the input high-resolution image into a low-resolution feature map, which is close to the shape and position of the object to be identified when extracting features. (42) The feature map after feature extraction is placed in the MaxViT block to apply attention, and the feature map is placed in the feature fusion module to fuse the global information. In each feature fusion process, the convolution layer is used to downsample the data to ensure that the size of the sampled data matches the size of the data processed by the MaxViT block; (43) The results processed by the four MaxViT blocks are added to the results processed by the feature fusion module, and the classification results are output after being processed by the pooling layer and the fully connected layer.

9. The switch cabinet fault diagnosis method based on thermal imaging and voiceprint according to claim 1 is characterized in that: The multimodal orthogonal sequence fusion method in step (5) merges the data step by step in order, and selectively weights the thermal imaging data with fault characteristics, the switch cabinet operation site pictures and voiceprint data, and the historical operation data; Train a single-modal model, rank the modalities according to their respective performance, start fusing from the worst performing modality and gradually move towards the best performing modality; use an orthogonal loss function to improve the fusion performance and interpretability of the model; The orthogonal loss function is expressed as: Among them, L is the loss function of a given task T, preds and outputs are the predictions and true values ​​of the model, respectively, and λ 1,i and λ 2,i are hyperparameters for the strength of the L1 and L2 regularization terms. The first term forces the absolute value of the dot product of the embedding vector to be close to zero, and the second term encourages the square of the dot product to be close to zero. Selective weighting is performed by mapping into the weight vector: Among them, a i is the weight of the ith mode, v i is the eigenvector of the ith mode; The diagnostic analysis method is as follows: a DC bias magnetic fault is identified through a voiceprint recognition model. When a discharge fault occurs, the voiceprint recognition technology captures these signals; by comparing historical operation data, it is determined whether it is within a threshold range and fault diagnosis is performed; characteristic information of thermal imaging and on-site images is detected to diagnose whether a fault exists and determine the fault location; if a fault exists, an alarm module is activated to remind staff to perform maintenance.

10. A switch cabinet fault diagnosis system based on thermal imaging and voiceprint, characterized in that: include: Data acquisition module: used to collect historical operation data, voiceprint data, thermal imaging data and switch cabinet operation site images; Data processing module: used to perform noise reduction processing on the collected voiceprint data and historical operation data; Model training module: used to predict the voiceprint set and historical operation data set using the time-enhanced spatiotemporal attention hybrid expert model, and build an improved MaxViT-DF model to train thermal imaging data and switch cabinet operation site images; Diagnostic module: When a discharge fault occurs, the fault signal is captured through voiceprint recognition technology, and historical operation data is used to determine whether it is within the threshold range. By detecting the characteristic information of thermal imaging and on-site images, it is diagnosed whether there is a fault and determines the fault location; Alarm module: used to activate the alarm module to remind staff to perform maintenance when there is a fault.

Images