A Power Equipment Anomaly Detection Method Based on Multimodal Deep Learning

Abstract

This invention discloses a method for power equipment anomaly detection based on multimodal deep learning, belonging to the field of power equipment anomaly detection technology. The method includes the following steps: S1: Constructing an original multimodal dataset of power equipment encompassing audio data, vibration signals, and images; S2: Preprocessing the original multimodal dataset to obtain a preprocessed multimodal dataset; S3: Constructing a power equipment anomaly detection model based on multimodal deep learning; S4: Dividing the preprocessed multimodal dataset, then training and validating the power equipment anomaly detection model; S5: Applying the trained power equipment anomaly detection model to detect power equipment. This invention, by combining diverse features from audio data, vibration signals, and power equipment images, can comprehensively capture diverse anomaly behaviors of power equipment, adapting to anomaly detection under different operating conditions.

Description

Technical Field

[0001] This invention relates to the field of power equipment anomaly detection technology, specifically a power equipment anomaly detection method based on multimodal deep learning. Background Technology

[0002] During long-term operation, power equipment may experience abnormalities due to aging, damage, or external factors. Timely detection and accurate diagnosis of these abnormalities are crucial for preventing failures, improving equipment reliability, and ensuring power supply safety. Traditional detection methods typically rely on manual inspections or sensor-based data monitoring, but these methods face challenges: manual inspections are inefficient and lack comprehensive coverage; traditional sensors cannot provide sufficient multi-dimensional information, making it difficult to detect potential problems in a timely manner; and manual inspections are affected by subjective factors and cannot provide long-term, real-time monitoring.

[0003] In recent years, with the development of artificial intelligence technology, especially the breakthroughs in deep learning, power equipment anomaly detection has ushered in new opportunities. Deep learning models, particularly convolutional neural networks (CNNs) and recurrent neural networks (RNNs), have demonstrated powerful capabilities in complex pattern recognition and feature extraction. By training deep learning models, potential anomalies can be automatically detected based on historical data and real-time sensor data, and the equipment status can be accurately predicted. This data-driven approach is more efficient and accurate than traditional manual inspection, providing real-time early warning information and improving the level of intelligent maintenance of power equipment.

[0004] However, existing methods still face challenges. The operating status of power equipment is affected by a variety of factors, and abnormal signals may be weak and similar to normal conditions, making them easy to overlook. Furthermore, power equipment data has multimodal characteristics, often requiring the fusion of information from different sensors and data sources to comprehensively assess the equipment's status. Existing deep learning models may suffer from reduced accuracy when processing multimodal data, especially in cases of data noise, missing data, or imbalance.

[0005] Therefore, improving the accuracy and reliability of power equipment anomaly detection, especially the processing of complex multimodal data, has become a research hotspot. Multimodal deep learning-based methods can integrate different data sources, identify potential anomalies, and maintain high accuracy and robustness even with incomplete data or noise interference. By optimizing model structure and training strategies, the level of automation and intelligence in detection can be effectively improved, providing strong support for the safe operation of power systems.

[0006] However, existing power equipment anomaly detection models based on multimodal deep learning have the following problems:

[0007] 1. Inability to comprehensively capture diverse abnormal behaviors of power equipment: Power equipment may experience different types of faults during operation (such as overload, short circuit, mechanical damage, etc.). These faults exhibit diverse abnormal behaviors and vary in time and space. Existing models fail to effectively identify all abnormal situations, resulting in insufficient comprehensiveness and accuracy of detection.

[0008] 2. Inability to accurately extract abnormal features from audio data, vibration signals, or images of power equipment: Existing models cannot fully exploit potential abnormal features when processing data of different modes, resulting in low accuracy in identifying equipment faults.

[0009] 3. Information imbalance between modalities: The inability to reasonably and effectively utilize the information of each model and assign appropriate weights to the models can easily lead to bias towards a certain modality, affecting the stability and accuracy of the final detection results.

[0010] Based on the above, a method for detecting anomalies in power equipment based on multimodal deep learning is invented. Summary of the Invention

[0011] To address the aforementioned technical problems, according to one aspect of the present invention, the present invention provides the following technical solution:

[0012] The power equipment anomaly detection method based on multimodal deep learning includes the following specific steps:

[0013] S1: Construct a raw multimodal dataset of power equipment that includes audio data, vibration signals, and images of power equipment;

[0014] S2: Preprocess the original power equipment multimodal dataset to obtain the preprocessed power equipment multimodal dataset;

[0015] S3: Construct an anomaly detection model for power equipment based on multimodal deep learning;

[0016] S4: Divide the preprocessed multimodal dataset of power equipment, and then train and validate the power equipment anomaly detection model;

[0017] S5: Apply the trained power equipment anomaly detection model to detect power equipment.

[0018] As a preferred embodiment of the power equipment anomaly detection method based on multimodal deep learning described in this invention, the specific steps of step S1 are as follows:

[0019] S11: First, use a sound sensor to collect audio data generated by the power equipment during operation and save it as an audio file; then, use a vibration sensor to collect vibration signals of the power equipment and save them as vibration data files; next, use a high-definition camera to capture images of the working area of ​​the power equipment, record images of the power equipment covering key components, and save them as image files.

[0020] S12: Label the acquired audio files, vibration data files, and image files; determine whether the current device has a fault from the audio data in the audio files, and then label the audio data as either faulty or fault-free; determine whether the current device has a fault from the vibration signals in the vibration data files, and then label the vibration signals as either faulty or fault-free; determine whether the current device has a fault from the power equipment images in the image files, and then use the image labeling tool to label the power equipment images as either faulty or fault-free.

[0021] As a preferred embodiment of the power equipment anomaly detection method based on multimodal deep learning described in this invention, the specific steps of S2 are as follows:

[0022] S21: Normalize the audio data in the audio file, that is, unify the sampling frequency and sampling duration, so that the audio data in the audio file has the same sampling interval, and obtain a one-dimensional sound subset.

[0023] S22: Process the audio data in the audio file into a Mel spectrogram using Mel frequency cepstral coefficients to obtain a two-dimensional sound subset;

[0024] S23: Normalize the vibration signals in the vibration data file to make their amplitudes within a uniform range, and then use the sliding window method to divide the vibration data into multiple time periods, each containing vibration data of uniform length, to obtain a one-dimensional vibration subset.

[0025] S24: Convert the vibration signals in the vibration data file into Gram angular field diagrams to obtain a two-dimensional vibration subset data set;

[0026] S25: Adjust the power equipment images in the image file to a uniform size, then use data augmentation techniques to generate multiple augmented images and add them to the dataset to obtain an image subset;

[0027] S26: Perform time alignment: For each sampling time point, extract the corresponding data samples from all subsets, including normalized audio data from the one-dimensional sound subset, Mel spectrograms from the two-dimensional sound subset, normalized vibration signals from the one-dimensional vibration subset, Gram angular field maps from the two-dimensional vibration subset, and images of power equipment from the image subset, and combine them to form a quintuple sample; the process achieves cross-modal synchronization through timestamp matching to ensure that the data samples of all subsets are aligned in the time dimension; repeat this alignment operation for all available time points to obtain the preprocessed multimodal dataset of power equipment.

[0028] As a preferred embodiment of the power equipment anomaly detection method based on multimodal deep learning described in this invention, the specific steps of S3 are as follows:

[0029] S31, Anomaly Detection Based on Power Equipment Audio Data: First, the Mel spectrogram from the two-dimensional sound subset of S2 is used to extract the frequency domain features of the audio data using MSFE-Net; then, the normalized audio data from the one-dimensional sound subset of S2 is used to extract the temporal domain features of the audio data using BiLSTM; subsequently, the results of the two are fused using time-frequency co-feature at the feature layer to obtain the audio feature map of the power equipment audio data; then, the audio feature map of the power equipment audio data is classified using an MLP classifier; finally, the classification results are converted into audio prediction of fault probability and audio prediction of no fault probability using a Softmax activation function.

[0030] S32, Anomaly Detection Based on Vibration Signals of Power Equipment: First, the Gram angular field map from the two-dimensional vibration subset of S2 is used to extract the spatial features of the vibration signal using GAFE-Net; then, the normalized vibration signal from the one-dimensional vibration subset of S2 is used to extract the temporal features of the vibration signal using BiGRU; subsequently, the results of the two are fused at the feature layer to obtain the vibration feature map of the power equipment vibration signal; then, the vibration feature map of the power equipment vibration signal is classified using an MLP classifier; finally, the classification results are converted into the vibration prediction probability of fault and the vibration prediction probability of no fault using a Softmax activation function.

[0031] S33, Anomaly detection based on power equipment images: For power equipment images in image files, ADEI-Net is used to perform target detection on the power equipment images, extract the visual features of the equipment, and determine the anomalies in the images, outputting the probability of the image being predicted to have a fault and the probability of the image being predicted to be fault-free.

[0032] S34, Determine if the power equipment is faulty: Combine the predicted probabilities of each mode, and then perform decision-level fusion through a multimodal weighted fusion algorithm to obtain the overall predicted probability of fault and the overall predicted probability of no fault; and determine whether the power equipment is faulty based on the probability values; if the overall predicted probability of fault is greater than the overall predicted probability of no fault, then the output equipment is faulty; otherwise, the output equipment is faultless.

[0033] As a preferred embodiment of the power equipment anomaly detection method based on multimodal deep learning described in this invention, the specific steps of S31 are as follows:

[0034] S311: Extract the frequency domain features of the audio data from the Mel spectrograms in the two-dimensional sound subset using the Mel spectrogram feature extraction network MSFE-Net, and obtain the Mel high-level feature map F. 19 ;

[0035] S312: Extracting Temporal Features from Audio Data: The normalized audio data from the one-dimensional sound subset is input into a Bidirectional Long Short-Term Memory (BiLSTM) network. First, the forward LSTM layer in the BiLSTM processes the audio data from left to right step by step, progressively updating the forward hidden state to capture the forward dependencies of the audio data in the time series. Simultaneously, the backward LSTM layer in the BiLSTM processes the audio data from right to left step by step, updating the backward hidden state to capture the backward dependencies of the audio data. Next, the forward and backward hidden states are concatenated at each time step to obtain the bidirectional features of that time step. Finally, global average pooling is performed on the bidirectional features of all time steps to obtain the temporal feature map E1.

[0036] S313: F 19 The audio feature map of the power equipment audio data is obtained by concatenating E1 along the feature dimension. Then, an MLP classifier is used to classify the audio feature map of the power equipment audio data, resulting in a two-dimensional vector containing the audio fault logits score and the audio non-fault logits score. Finally, the two-dimensional vector is normalized using the Softmax activation function to obtain the audio fault prediction probability. And audio prediction fault-free probability

[0037] The specific steps of S311 are as follows:

[0038] S3111: Represent any Mel spectrogram in the two-dimensional sound subset as F1, input it into a convolutional layer with 64 3×3 kernels and a stride of 1, and then pass it through the ReLU activation function and batch normalization to obtain the Mel shallow feature map F2; then, input F2 into a convolutional layer with 64 3×3 kernels and a stride of 1, and then pass it through the ReLU activation function and batch normalization to obtain the Mel shallow feature map F3;

[0039] S3112: Input F3 into the SAPM module to obtain the Mel shallow feature map F4; where F3 is denoted as A1;

[0040] S3113: Input F4 into a convolutional layer with 128 3×3 kernels and a stride of 1, then pass it through the ReLU activation function and batch normalization to obtain the Mel shallow feature map F5; then input F5 into a convolutional layer with 128 3×3 kernels and a stride of 1, then pass it through the ReLU activation function and batch normalization to obtain the Mel shallow feature map F6; finally, input F6 into the SAPM module to obtain the Mel shallow feature map F7;

[0041] S3114: Input F7 into a convolutional layer with 256 3×3 kernels and a stride of 1, then pass it through ReLU activation and batch normalization to obtain the Mel-Ray mid-layer feature map F8; then input F8 into a convolutional layer with 256 3×3 kernels and a stride of 1, then pass it through ReLU activation and batch normalization to obtain the Mel-Ray mid-layer feature map F9; finally, input F9 into the SAPM module to obtain the Mel-Ray mid-layer feature map F... 10 ;

[0042] S3115: F 10 The input is fed into a convolutional layer with 512 3×3 kernels and a stride of 1, and then sequentially processed by ReLU activation and batch normalization to obtain the Mel high-level feature map F. 11 Then F 11 The input is fed into a convolutional layer with 512 3×3 kernels and a stride of 1, and then sequentially processed by ReLU activation and batch normalization to obtain the Mel high-level feature map F. 12 ;

[0043] S3116: F 12 The input is fed into the LGFA module to obtain the Mel high-level feature map F. 13 ;

[0044] S3117: F 13 The input is fed into the SAPM module to obtain the Mel high-level feature map F. 14 Next, F 14The input is fed into a convolutional layer with 512 3×3 kernels and a stride of 1, and then sequentially processed by ReLU activation and batch normalization to obtain the Mel high-level feature map F. 15 Then, F 15 The input is fed into a convolutional layer with 512 3×3 kernels and a stride of 1, and then sequentially processed by ReLU activation and batch normalization to obtain the Mel high-level feature map F. 16 After that, F 16 The input is fed into the LGFA module to obtain the Mel high-level feature map F. 17 Next, F 17 The input is fed into the SAPM module to obtain the Mel high-level feature map F. 18 Finally, F 18 The input is fed into a fully connected layer, and the high-level feature map F of Mel is obtained and output. 19 ;

[0045] The specific steps of S3112 are as follows:

[0046] S31121: In the first branch, calculate A along the vertical axis. 1_p The mean value is used to obtain the vertical mean vector μ. p,h In the second branch, calculate A along the vertical axis. 1_p The variance is used to obtain the vertical variance vector.

[0047] μ p,h =[μ p,h [1],μ p,h [2],…,μ p,h [h]

[0048]

[0049] In the above formula, A 1_p (i,j) represents the state in A. 1_p The eigenvalue at position (i,j) in the middle;

[0050] S31122: In the third branch, according to A 1_p Calculate the height H to output the height H 2_p ;

[0051]

[0052] S31123: In the fourth branch, A is not correct. 1_p Perform any operation; in the fifth branch, calculate A along the horizontal axis. 1_p The mean value is used to obtain the water average vector μ. p,w In the sixth branch, calculate A along the horizontal axis. 1_p The variance is used to obtain the level variance vector.

[0053] μ p,w =[μ p,w [1],μ p,w [2],…,μ p,w [w]

[0054]

[0055]

[0056] In the above formula, A 1_p (i,j) represents the state in A. 1_p The eigenvalue at position (i,j) in the middle;

[0057] S31124: In the seventh branch, according to A 1_p The width W is used to calculate the output width W. 2_p ;

[0058]

[0059] S31125: Based on variance and Calculations are performed to obtain the variance factor. Then according to μ p,h Variance-guided fusion calculation was performed to obtain the mean μ. h Then according to Will Variance-guided fusion calculation is performed to obtain the variance. Then according to μ p,w Variance-guided fusion calculation was performed to obtain the mean μ. w Then according to Will Variance-guided fusion calculation was performed to obtain the mean. Subsequently, according to μ h and The calculation is performed to obtain the vertical pooling window size k. h According to μ w and The calculation is performed to obtain the horizontal pooling window size k. w ;

[0060]

[0061] In the above formula, max() represents the operation of calculating the maximum value; mean() represents the operation of calculating the average value; round() represents the operation of rounding the number to the nearest integer; k minε is the size of the minimum pooling window, used to avoid the pooling window being too small; ε is a constant used to prevent instability caused by the mean being close to zero or the variance being too small.

[0062] S31126: According to k h and H 2_p Calculate the vertical step size s h Then, according to k w and W 2_p Calculate the horizontal step size s w ;

[0063]

[0064] In the above formula, round() means rounding the number to the nearest integer.

[0065] S31127: According to A 1_p Height H, Output Height H 2_p Vertical pooling window size k h and vertical step size s h Calculate the vertical padding amount. h According to A 1_p Width W, Output Width W 2_p Horizontal pooling window size k w and horizontal step size s w Calculate the horizontal padding amount. w ;

[0066]

[0067] Next, for A 1_p Perform zero-fill on all four sides sequentially: first in A 1_p Add padding to the left w Fill the column with zero, then add padding on the right. w Fill the column with zero, then add padding at the top. h Use zero padding in the rows, then add padding at the bottom. h Row zero padding; ultimately resulting in a size of (H + 2 * padding) h )×(W+2*padding w The feature map of the four sides with zero fill;

[0068] Finally, based on the calculated pooling window size k h k w and step size s h s wMax pooling is performed on the feature map with zero padding on all four sides to obtain a single-channel feature map A of size H / 2×W / 2. 2_p ;

[0069] The specific steps of S3116 are as follows:

[0070] S31161: In the first branch, no operation is performed on B1; in the second branch, B1 is first subjected to global average pooling to obtain local attention feature map B2; a fully connected layer is used to reduce the dimensionality of B2 to obtain local attention feature map B3; then B3 is passed through the ReLU activation function to obtain local attention feature map B4; then another fully connected layer is used to increase the dimensionality of B4 to obtain local attention feature map B5, making the number of channels of B5 the same as that of B2; then B5 is passed through the Sigmoid activation function to obtain local attention feature map B6; then B6 is multiplied element-wise with B1 to obtain local attention feature map B7; subsequently, B7 is first subjected to global average pooling, then a multilayer perceptron is used to transform the pooling result, and the transformed result is passed through the Sigmoid activation function to obtain local attention feature map B8; finally, B7 and B8 are multiplied element-wise to obtain local attention feature map B9.

[0071] B7=B1⊙S(FC2(ReLU(FC1(GAP(B1)))))

[0072] B9 = B7⊙S(MLP(GAP(B7)))

[0073] In the above formula, S represents the Sigmoid activation function; GAP() represents the Global Avg Pool operation; ReLU represents the ReLU activation function; MLP() represents multilayer perceptron processing; FC1() represents dimensionality reduction processing through a fully connected layer; FC2() represents dimensionality increase processing through a fully connected layer; ⊙ represents element-wise multiplication operation.

[0074] S31162: In the third branch, the weight matrix W v Multiplying by B1 yields the value matrix V; in the fourth branch, the weight matrix W is... q Multiplying by B1 yields the query matrix Q; in the fifth branch, the weight matrix W is... k Multiplying Q by B1 yields the key matrix K; then, the dot product of Q and K is calculated, and the attention score matrix S is obtained by applying the Softmax activation function; finally, S is multiplied by V to obtain the global attention feature map B. 10 Next, a 1×1 convolution kernel is applied to B. 10 Perform a convolution operation to obtain the global attention feature map B. 11 Then, for B 11Global average pooling is performed, followed by feature transformation of the pooling result using a multilayer perceptron. The transformed feature result is then passed through a sigmoid activation function to obtain the global attention feature map B. 12 Subsequently, B 11 With B 12 Perform element-wise multiplication to obtain the global attention feature map B. 13 ;

[0075] V = W v ×B1

[0076] Q = W q ×B1

[0077] K = W k ×B1

[0078]

[0079] B 12 =S(MLP(GAP(B) 11 )))

[0080] B 13 =B 11 ⊙B 12

[0081] In the above formula, S represents the Sigmoid activation function; Softmax represents the Softmax activation function; GAP() represents the Global Avg Pool operation; MLP() represents the multilayer perceptron processing; and the He initialization method is used to initialize the weight matrix W. q W k and W v Perform initialization, W q W k and W v All dimensions are Where C' is the number of channels after compression; × represents matrix multiplication; ⊙ represents element-wise multiplication; + represents element-wise addition;

[0082] S31163: Finally, connect B9 and B 13 Perform element-wise addition to obtain and output the fused attention feature map B. 14 .

[0083] As a preferred embodiment of the power equipment anomaly detection method based on multimodal deep learning described in this invention, the specific steps of S32 are as follows:

[0084] S321: Extract the spatial features of the vibration signal from the Gram angular field map in the two-dimensional vibration subset using the Gram angular field map feature extraction network GAFE-Net, and obtain the Gram extracted feature map G.15 ;

[0085] S322: Extracting Temporal Features of Vibration Signals: The normalized vibration signals from the one-dimensional vibration subset are input into a bidirectional gated recurrent unit (BiGRU). First, the forward GRU layer in the BiGRU processes the vibration signals from left to right in chronological order, using a gating mechanism containing only update and reset gates to progressively update the forward hidden state and capture the forward temporal pattern of the vibration signals. Simultaneously, the reverse GRU layer processes the signals in reverse order from right to left, controlling the information flow through gating to capture the reverse dependencies of the vibration signals. Then, at each time step, the forward and reverse hidden states are concatenated along the channel dimension to form a fused feature containing bidirectional temporal information. Finally, global average pooling is performed on the fused features of all time steps to obtain the temporal feature map N1.

[0086] S323: G 15 The vibration feature map of the power equipment vibration signal is obtained by concatenating N1 with N1 along the feature dimension. Then, the vibration feature map of the power equipment vibration signal is classified by an MLP classifier to obtain a two-dimensional vector containing the logits scores of vibration with and without faults. Finally, the two-dimensional vector is normalized by the Softmax activation function to obtain the vibration prediction probability of faults. And vibration prediction fault-free probability

[0087] The specific steps of S321 are as follows:

[0088] S3211: Represent any Gram angular field map from the two-dimensional vibration subset as G1, input it into a convolutional layer with a kernel size of 4×4 and a stride of 4, and then perform layer normalization to obtain the Gram extracted feature map G2; then, pass G2 through 3 consecutive Bottleneck modules to obtain the Gram extracted feature map G3; then, pass G3 through a Downsample module to reduce its height by half, width by half, and keep the number of channels unchanged to obtain the Gram extracted feature map G4. Specifically, first, pass G3 through layer normalization, then pass the normalized result through a convolutional layer with a kernel size of 2×2 and a stride of 2 to obtain G4; then, pass G4 through 5 consecutive Bottleneck modules to obtain the Gram extracted feature map G5.

[0089] S3212: Input G5 into the DAACO module to obtain Gram extracted feature map G6;

[0090] S3213: Input G6 into the Downsample module to reduce its height and width by half while keeping the number of channels unchanged, to obtain the Gram extraction feature map G7; then, pass G7 through 7 Bottleneck modules to obtain the Gram extraction feature map G8; then, pass G8 through the DAACO module to obtain the Gram extraction feature map G9.

[0091] S3214: Input G9 into the MLAF module to obtain the Gram extracted feature map G. 10 ;

[0092] S3215: G 10 The input is fed into the Downsample module, which reduces its height and width by half while keeping the number of channels unchanged, resulting in the Gram extracted feature map G. 11 Then, G 11 The Gram feature map G is obtained by passing through 5 Bottleneck modules consecutively. 12 After that, G 12 The Gram feature map G is obtained through the MLAF module. 13 Next, G 13 Perform global average pooling to obtain the Gram extracted feature map G. 14 Finally, G 14 Perform layer normalization to obtain and output the Gram extracted feature map G. 15 ;

[0093] The specific steps of S3212 are as follows:

[0094] S32121: In the first branch, no operation is performed on L1; in the second branch, global average pooling is used to pool L1 to obtain the channel-optimized feature map L2; in the third branch, a pooling window of size 3×3 and a dynamic step size s are first applied. D Averaging is performed on L1, and then global averaging is performed on the result to obtain the channel-optimized feature map L3. In the fourth branch, a pooling window of size 5×5 and a dynamic step size s are first applied. D Perform average pooling on L1, and then perform global average pooling on the result to obtain the channel-optimized feature map L4.

[0095]

[0096] L2 = GAP(L1)

[0097] L3 = GAP(AvgPool3(L1))

[0098] L4 = GAP(AvgPool5(L1))

[0099] In the above formula, It is a hyperparameter; max() indicates the maximum value calculation operation; GAP() indicates the Global AvgPool operation; AvgPool3() indicates a pooling window size of 3×3 and a stride of s. D Average pooling operation; AvgPool5() represents a pooling window size of 5×5 with a stride of s. D Average pooling operation;

[0100] S32122: Adjust the channel sparse parameter W S1 Multiplying by L2 yields the channel-optimized feature map L5; the channel sparsity parameter W is then... S2 Multiplying by L3 yields the channel-optimized feature map L7; the channel sparsity parameter W is then... S3 Multiplying by L4 yields the channel-optimized feature map L9; then, element-wise addition of L5, L7, and L9 yields the channel-optimized feature map L0. 11 Then L 11 After applying the Sigmoid activation function, the channel-optimized feature map L is obtained. 12 Next, L 12 Element-wise multiplication with L1 yields the channel-optimized feature map L. 13 ;

[0101] L 13 =S(W S1 ×L2+W S2 ×L3+W S3 ×L4)⊙L1

[0102] In the above formula, the Xavier initialization method is used to initialize the channel sparse parameter W. S1 W S2 and W S3 Initialize; S represents the Sigmoid activation function; × represents matrix multiplication; ⊙ represents element-wise multiplication; + represents element-wise addition;

[0103] S32123: Transfer abnormal channel parameter W A1 Multiplying by L2 yields the channel optimization feature map L6; the abnormal channel parameters W are then... A2 Multiplying by L3 yields the channel optimization feature map L8; the abnormal channel parameters W are then... A3 Multiplying by L4 yields the channel-optimized feature map L. 10 Then add L6, L8 and L 10 Perform element-wise addition to obtain the channel-optimized feature map L. 14 Then L 14 After applying the ReLU activation function, the channel-optimized feature map L is obtained. 15;

[0104] L 15 =ReLU(W A1 ×L6+W A2 ×L8+W A3 ×L 10 )

[0105] The above formula uses a Gaussian distribution. The initialization method addresses the abnormal channel parameter W. A1 W A2 and W A3 Initialize; ReLU represents the ReLU activation function; × represents matrix multiplication; + represents element-wise addition.

[0106] S32124: L 13 With L 15 Perform element-wise multiplication to obtain the channel-optimized feature map L. 16 Then, the channel attention parameter W D With L 16 Perform a multiplication operation to obtain the channel-optimized feature map L. 17 Then L 17 After applying the Softmax activation function, the channel-optimized feature map L is obtained. 18 Finally, L 16 With L 18 Perform element-wise multiplication to obtain and output the channel-optimized feature map L. 19 ;

[0107] L 19 =L 16 ⊙Softmax(W D ×(L 13 ⊙L 15 ))

[0108] The above formula uses a Gaussian distribution. Initialization method, for channel attention parameter W D Initialize; Softmax represents the Softmax activation function; × represents matrix multiplication; ⊙ represents element-wise multiplication.

[0109] The specific steps of S3214 are as follows:

[0110] S32141: In the first branch, no operation is performed on T1; in the second branch, no operation is performed on T1; in the third branch, wavelet transform is first applied to T1 to extract high-frequency anomaly features, resulting in a multi-level fusion feature map T2; then, a 1×1 convolution kernel is applied to T2 to obtain a multi-level fusion feature map T3; finally, T3 is processed through a Sigmoid activation function to obtain a multi-level fusion feature map T4.

[0111] T4 = S(Conv1(DWT(T1)))

[0112] In the above formula, DWT() represents the wavelet transform operation; Conv1() represents the convolution operation with a kernel of size 1×1; and S represents the sigmoid activation function.

[0113] S32142: In the fourth branch, a depthwise separable convolution operation is performed on T1 using a 1×1 convolution kernel to obtain a multi-level fused feature map T5; in the fifth branch, a depthwise separable convolution operation is performed on T1 using a 3×3 convolution kernel to obtain a multi-level fused feature map T6; in the sixth branch, a depthwise separable convolution operation is performed on T1 using a 5×5 convolution kernel to obtain a multi-level fused feature map T7; in the seventh branch, a depthwise separable convolution operation is performed on T1 using a 7×7 convolution kernel to obtain a multi-level fused feature map T8.

[0114] Next, T5, T6, T7, and T8 are added element-wise to obtain the multi-level fused feature map T9; then, T9 is activated by the Sigmoid activation function to obtain the multi-level fused feature map T. 10 :

[0115] T 10 =S(DSC1(T1)+DSC3(T1)+DSC5(T1)+DSC7(T1))

[0116] In the above formula, DSC1() represents a depthwise separable convolution operation with a 1×1 kernel; DSC3() represents a depthwise separable convolution operation with a 3×3 kernel; DSC5() represents a depthwise separable convolution operation with a 5×5 kernel; DSC7() represents a depthwise separable convolution operation with a 7×7 kernel; + represents element-wise addition; S represents the Sigmoid activation function.

[0117] S32143: In the eighth branch, average pooling is performed on T1 along the channel dimension. Specifically, it iterates through each spatial position (i,j) of T1 (where i∈[1,H], j∈[1,W]) and calculates the average value of its position across all channels. In the ninth branch, max pooling is performed on T1 along the channel dimension. Specifically, it iterates through each spatial position (i,j) of T1 (where i∈[1,H], j∈[1,W]) and calculates the maximum value of its position across all channels. The results from the eighth and ninth branches are then concatenated along the channel dimension to obtain the multi-level fused feature map T. 11 Subsequently, a 1×1 convolution kernel was applied to T. 11 Perform convolution operations to obtain a multi-level fused feature map T. 12 Then apply a 7×7 convolution kernel to T 12 Perform convolution operations to obtain a multi-level fused feature map T. 13 Using a 3×3 convolution kernel with a dilation rate of 2 on T 12 Perform convolution operations to obtain a multi-level fused feature map T. 14 Then T 13 and T 14 Perform element-wise addition to obtain the multi-level fused feature map T. 15 ; then T 15 After applying the Sigmoid activation function, a multi-level fused feature map T is obtained. 16 :

[0118]

[0119] T 15 =S(Conv7(T) 12 )+Conv3_2(T 12 ))

[0120] In the above formula, T1(i,j,k) represents the feature value of T1 at spatial position (i,j) and the k-th channel; max k∈[1,C] () indicates the operation of calculating the maximum value of each spatial location across all channels; Concat() indicates the concatenation operation along the channel dimension; Conv7() indicates the convolution operation with a 7×7 kernel; Conv3_2() indicates the convolution operation with a 3×3 kernel with a dilation of 2; S indicates the Sigmoid activation function; + indicates element-wise addition.

[0121] S32144: Place T1, T4, T 10 and T 16 Perform element-wise multiplication to obtain the multi-level fused feature map T. 17 Then apply a 1×1 convolution kernel to T.17 Perform convolution operations to obtain a multi-level fused feature map T. 18 Then T 18 After applying the Softmax activation function, the anomaly mask M is obtained. a ;

[0122] After that, for M a Perform two sub-branch calculations; in the first sub-branch, calculate M. a The inverse anomaly mask is obtained by using the inverse anomaly mask M. 1-a Then T1 and M 1-a Perform element-wise multiplication to obtain the multi-level fused feature map T. 19 In the second sub-branch, T 17 and M a Perform element-wise multiplication to obtain the multi-level fused feature map T. 20 Finally, T 19 and T 20 Perform element-wise addition to obtain and output the multi-level fused feature map T. 21 ;

[0123] M a =Softmax(Conv1(T1⊙T4⊙T) 10 ⊙T 16 ))

[0124] M 1-a =1-M a

[0125] T 21 =T1⊙M 1-a +M a ⊙T 17

[0126] In the above formula, Conv1() represents a convolution operation with a kernel of size 1×1; Softmax represents the Softmax activation function; ⊙ represents element-wise multiplication; + represents element-wise addition.

[0127] As a preferred embodiment of the power equipment anomaly detection method based on multimodal deep learning described in this invention, the specific steps of S33 are as follows:

[0128] S331: Represent any power equipment image in the image subset as P1, input it into the CBS module to obtain the power equipment detection feature map P2; then, pass P2 through the CBS module to obtain the power equipment detection feature map P3; then, pass P3 through the C2F module to obtain the power equipment detection feature map P4; then, pass P4 through the CBS module to obtain the power equipment detection feature map P5.

[0129] S332: Input P5 into the CFAE module to obtain the power equipment detection feature map P6;

[0130] S333: Input P6 into the CBS module to obtain the power equipment detection feature map P7; then, pass P7 through the CFAE module to obtain the power equipment detection feature map P8; next, pass P8 through the CBS module to obtain the power equipment detection feature map P9; finally, pass P9 through the CFAE module to obtain the power equipment detection feature map P... 10 Then P 10 The SPPF module is used to obtain the power equipment detection feature map P. 11 Subsequently, P 11 An upsampling operation is performed to obtain the power equipment detection feature map P. 12 Next, P8 and P 12 By stitching along the channel dimension, the power equipment detection feature map P is obtained. 13 Then P 13 The power equipment detection feature map P is obtained through the C2F module. 14 ; then P 14 An upsampling operation is performed to obtain the power equipment detection feature map P. 15 Next, P6 and P 15 By stitching along the channel dimension, the power equipment detection feature map P is obtained. 16 Then P 16 The power equipment detection feature map P is obtained through the C2F module. 17 ; then P 17 The power equipment detection feature map P is obtained through the CBS module. 18 Next, P 14 and P 18 By stitching along the channel dimension, the power equipment detection feature map P is obtained. 19 Then, P 19 The power equipment detection feature map P is obtained through the C2F module. 20 After that, P 20 The power equipment detection feature map P is obtained through the CBS module. 21 Then P 11 and P 21 By stitching along the channel dimension, the power equipment detection feature map P is obtained. 22 Next, P 22 The power equipment detection feature map P is obtained through the C2F module. 23 ;

[0131] S334: P 17 The data is input into the AADM module to obtain the power equipment detection feature map P. 24 ; P20 The data is input into the AADM module to obtain the power equipment detection feature map P. 27 ; P 23 The data is input into the AADM module to obtain the power equipment detection feature map P. 30 ;

[0132] S335: P 24 The data is input into the AADM module to obtain the power equipment detection feature map P. 25 ; P 27 The data is input into the AADM module to obtain the power equipment detection feature map P. 28 ; P 30 The data is input into the AADM module to obtain the power equipment detection feature map P. 31 ;

[0133] After that, P 25 The power equipment detection feature map P is obtained by using a convolutional layer with a kernel size of 1×1. 26 ; P 28 The power equipment detection feature map P is obtained by using a convolutional layer with a kernel size of 1×1. 29 ; P 31 The power equipment detection feature map P is obtained by using a convolutional layer with a kernel size of 1×1. 32 ;

[0134] Subsequently, for P 26 The values ​​at all locations in the spatial dimension are averaged to obtain a two-dimensional vector X1 containing the faulty logits score and the fault-free logits score of the image; for P 29 The values ​​at all locations in the spatial dimension are averaged to obtain a two-dimensional vector X2 containing the faulty logits score and the fault-free logits score of the image; P 32 The values ​​at all locations in the spatial dimension are averaged to obtain a two-dimensional vector X3 containing the image faulty logits score and the image fault-free logits score. Then, the three levels of two-dimensional vectors are summed element-wise to obtain a comprehensive logits vector X4. Finally, the comprehensive logits vector is transformed into a probability distribution through a softmax activation function to obtain a two-dimensional vector P containing the image prediction fault probability and the image prediction fault-free probability. out Then according to P out The probability of failure is obtained from image prediction. And image prediction fault-free probability

[0135] X4 = X1 + X2 + X3

[0136] P out=Softmax(X4)

[0137]

[0138] In the above formula, Softmax represents the Softmax activation function; + indicates element-wise addition.

[0139] The specific steps of S332 are as follows:

[0140] S3321: Input the power equipment detection feature map V1. First, use the channel shuffling method to cross-mix the channel information of V1 to obtain the channel feature enhancement feature map V2. Then, use segmented GLU to perform gating processing on V2 to obtain the channel feature enhancement feature map V3. The specific steps are: first, divide V2 into two parts along the channel dimension V3. 2_1 and V 2_2 Then V 2_2 The gate signal G is obtained through the Sigmoid activation function, and finally V is... 2_1 Element-wise multiplication with G yields V3; then, V3 is convolved with a 1×1 kernel to obtain the channel feature enhancement feature map V4.

[0141] S3322: Perform four branch calculations on V4. In the first branch, first, use global average pooling to pool V4 to obtain the channel feature enhancement feature map V5; then, use a 1×1 convolution kernel to convolve V5 to obtain the channel feature enhancement feature map V6; then, pass V6 through the Sigmoid activation function to obtain the channel feature enhancement feature map V7. In the second branch, use a 3×3 convolution kernel to convolve V4 to obtain the channel feature enhancement feature map V8; then, pass V8 through the Tanh activation function to obtain the channel feature enhancement feature map V9. In the third branch, do not perform any operation on V4; then, multiply V7, V9, and V4 element-wise to obtain the channel feature enhancement feature map V. 11 ;

[0142] S3323: In the fourth branch, V4 is processed by the Bottleneck module for feature extraction to obtain the channel feature enhancement feature map V. 10 Finally, V 11 and V 10 Perform channel-wise maximum fusion to obtain and output channel feature enhancement feature map V. 12 The specific operation is as follows: For V 11 and V 10 For each corresponding channel, traverse all spatial locations on the feature map of that channel and calculate V. 11 and V 10 The maximum value of the element at that position, and use it as V.12 Elements in;

[0143] The specific steps of S334 are as follows:

[0144] S3341: Input the power equipment detection feature map Y1. First, perform convolution operations on Y1 using a 3×3 kernel, a 5×5 kernel, and a 7×7 kernel. Then, concatenate the three convolution results along the channel dimension to obtain the attention anomaly detection feature map Y2. Next, perform convolution operations on Y2 using a 1×1 kernel to obtain the attention anomaly detection feature map Y3. Then, process Y3 with the ReLU activation function and then process the result with the Hard Sigmoid activation function to obtain the attention anomaly detection feature map Y4.

[0145] Y2=Concat(Conv3(Y1),Conv5(Y1),Conv7(Y1))

[0146] Y4 = HardS(ReLU(Conv1(Y2)))

[0147] In the above formula, Conv3() represents a convolution operation with a 3×3 kernel; Conv5() represents a convolution operation with a 5×5 kernel; Conv7() represents a convolution operation with a 7×7 kernel; Concat() represents a concatenation operation along the channel dimension; Conv1() represents a convolution operation with a 1×1 kernel; ReLU represents the ReLU activation function; HardS represents the Hard Sigmoid activation function.

[0148] S3342: Perform element-wise multiplication of Y4 and Y1 to obtain the attention anomaly detection feature map Y5; then apply a depthwise separable convolution operation to Y5 using a 3×3 convolution kernel to obtain the attention anomaly detection feature map Y6; then pass Y6 through the Sigmoid activation function to obtain the attention anomaly detection feature map Y7; finally, perform element-wise multiplication of Y7 and Y5 to obtain the attention anomaly detection feature map Y8;

[0149] Y8=(Y4⊙Y1)⊙S(DSC3(Y5))

[0150] In the above formula, DSC3() represents a depthwise separable convolution operation with a 3×3 kernel; S represents the Sigmoid activation function; ⊙ represents element-wise multiplication.

[0151] S3343: Use a fully connected layer to perform feature mapping on Y8 to obtain the attention anomaly detection feature map Y9; then, process Y9 through the sigmoid activation function to obtain the attention anomaly detection feature map Yi. 10 Finally, Y 10 Element-wise multiplication with Y8 yields the attention anomaly detection feature map Y. 11 ;

[0152] Y 11 =S(FC(Y8))⊙Y8

[0153] In the above formula, S represents the Sigmoid activation function; FC() represents the feature mapping operation of the fully connected layer; and ⊙ represents the element-wise multiplication operation.

[0154] As a preferred embodiment of the power equipment anomaly detection method based on multimodal deep learning described in this invention, the specific steps of S34 are as follows:

[0155] The input sub-model M1 has a fault probability for audio prediction of power equipment. And audio prediction fault-free probability Sub-model M2 has a failure probability for vibration prediction of power equipment. And vibration prediction fault-free probability Sub-model M3 has a fault probability for image prediction of power equipment. And image prediction fault-free probability First, calculate the mutual information I(M1,M2) between M1 and M2, the mutual information I(M1,M3) between M1 and M3, and the mutual information I(M2,M3) between M2 and M3. Then, based on the mutual information calculation results, adjust the weights of each sub-model using a weighted average to obtain the adjusted weight w of model M1. adjust (1) M2 model weights w adjust (2) and the weights w of the M3 model adjust (3); Then apply the weighted voting method to calculate the probability P of a faulty weighted vote. vote (1) and the fault-free weighted voting probability P vote (2); Then, the soft voting method is applied to calculate the probability P of a faulty soft vote. soft (1) and the probability of fault-free soft voting P soft (2); Finally, the results of the weighted fusion of the weighted voting method and the soft voting method are used to obtain the final prediction result: the overall prediction probability of failure P. mmwf (1) and the overall predicted fault-free probability P mmwf (2);

[0156]

[0157] P mmwf (y)=α*P vote (y)+(1-α)*P soft (y)

[0158] In the above formula, I(M) i M j ) indicates the calculation of M i and M j Mutual information between them; y represents a binary classification value indicating whether there is a fault, where y=1 indicates a fault and y=2 indicates no fault; Submodel M i The predicted probability of category y; P(y) is the prior probability of category y, where P(y=1) equals the ratio of the number of faulty samples in the dataset to the total number of samples, and P(y=2) equals the ratio of the number of fault-free samples in the dataset to the total number of samples; w i Each sub-model M represents i Learnable weights; k represents the number of sub-models; w adjust (i) represents the sub-model M adjusted based on mutual information. i Normalized weights; 1(y=y pred,i ) represents the sub-model M i The indicator function for whether the prediction is for category y; α represents a learnable parameter used to balance the effects of weighted voting and soft voting.

[0159] As a preferred embodiment of the power equipment anomaly detection method based on multimodal deep learning described in this invention, the specific steps of step S4 are as follows:

[0160] S41: Divide the preprocessed multimodal dataset of power equipment in S2 into training set, validation set and test set according to a certain ratio to ensure that the samples in the training set, validation set and test set are independent and do not overlap; during the partitioning process, ensure that the class distribution of each subset is consistent with the overall dataset to avoid the negative impact of data imbalance on model training;

[0161] S42: A dynamically adjusted hyperparameter scheme is adopted: the basic training cycle is set to a certain number of rounds, and a progressive batch expansion strategy is implemented, that is, the initial batch size is set to a small value, and the batch size is gradually increased after a certain number of training cycles; the learning rate scheduling adopts the cyclic cosine annealing algorithm, starting from a low baseline value, and setting the cycle length according to the training requirements, so that the learning rate decreases smoothly during the training process, effectively improving the model convergence; the optimizer is NAdam, and gradient normalization constraints are set to prevent the gradient from being too large by limiting the L2 norm of parameter updates, thus ensuring the stability of the training process;

[0162] S43: During the training phase, the input samples in each batch are a quintuple, which includes: normalized audio data from the one-dimensional sound subset, Mel spectrograms from the two-dimensional sound subset, normalized vibration signals from the one-dimensional vibration subset, Gram angular field maps from the two-dimensional vibration subset, and images of power equipment from the image subset. Through the quintuple samples, the model can learn the correlation between different modalities. During feature extraction, a cross-modal contrastive loss function is used to optimize the model, enabling data from different modalities to be effectively aligned in the feature space, thereby improving multimodal learning capabilities.

[0163] S44: During the validation phase, a multi-dimensional evaluation protocol is adopted, and a full validation is performed every few batches. The validation process includes several test modes: standard data inference, noise interference test, and modality missing test. An adaptive early stopping mechanism is set up: when the coefficient of variation of the main evaluation metric remains below the preset value in several consecutive validations, the training is automatically terminated to ensure the convergence and stability of the model.

[0164] S45: During the testing phase, the model will undergo several rigorous tests on the test set, including: cross-device generalization evaluation, real-time stress testing, model stability, robustness, and performance evaluation under data noise interference; through testing, it is ensured that the model can maintain high reliability and accuracy in practical applications and meet the actual needs of power equipment anomaly detection.

[0165] As a preferred embodiment of the power equipment anomaly detection method based on multimodal deep learning described in this invention, the specific steps of S5 are as follows:

[0166] S51: Input the collected audio data, vibration signals and images of the power equipment into the power equipment anomaly detection model to obtain the overall predicted probability of failure and the overall predicted probability of no failure.

[0167] S52: Then make a judgment. If the overall predicted probability of failure is greater than the overall predicted probability of no failure, then the output device is faulty; otherwise, the output device is not faulty.

[0168] Compared with existing technologies:

[0169] This invention combines diverse features from audio data, vibration signals, and images of power equipment. It not only comprehensively captures various abnormal behaviors of power equipment, adapting to anomaly detection under different operating conditions, but also effectively extracts abnormal features from audio data, vibration signals, and images of power equipment. This ensures the full mining of potential abnormal information in different modal data, overcoming the problem of inaccurate identification of abnormal features. Furthermore, it ensures the reasonable fusion of information from various modalities, effectively avoiding the problem of information imbalance between modalities and improving the stability and accuracy of the model. Attached Figure Description

[0170] Figure 1 This is a schematic diagram of the process of the present invention;

[0171] Figure 2 This is a diagram illustrating the overall architecture of the power equipment anomaly detection model of the present invention.

[0172] Figure 3 This is a diagram showing the overall structure of the Mel spectrogram feature extraction network MSFE-Net of this invention;

[0173] Figure 4 This is an overall structural diagram of the spatial adaptive pooling module SAPM of the present invention;

[0174] Figure 5 This is an overall structural diagram of the single-channel spatial adaptive pooling module SSAPM of the present invention;

[0175] Figure 6 This is an overall structural diagram of the Local-Global Fusion Attention Module (LGFA) of this invention;

[0176] Figure 7 This is a diagram showing the overall structure of the Gram Angular Field Map Feature Extraction Network GAFE-Net of this invention.

[0177] Figure 8 This is an overall structural diagram of the DAACO dynamic anomaly perception channel optimization module of the present invention;

[0178] Figure 9 This is an overall structural diagram of the multi-level adaptive fusion module MLAF of the present invention;

[0179] Figure 10 This is an overall structural diagram of the ADEI-Net power equipment image anomaly detection network of the present invention;

[0180] Figure 11 This is an overall structural diagram of the Channel Feature Adaptive Enhancement Module (CFAE) of the present invention;

[0181] Figure 12 This is an overall structural diagram of the attention anomaly detection module AADM of the present invention;

[0182] Figure 13 This is a diagram showing the overall structure of the multimodal weighted fusion algorithm MMWF of this invention. Detailed Implementation

[0183] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0184] This invention provides a method for detecting anomalies in power equipment based on multimodal deep learning. Please refer to [link / reference]. Figures 1-13 The specific steps are as follows:

[0185] S1: Construct a raw multimodal dataset of power equipment that includes audio data, vibration signals, and images of power equipment;

[0186] The specific steps of S1 are as follows:

[0187] S11: First, use a sound sensor to collect audio data generated by the power equipment during operation and save it as an audio file; then, use a vibration sensor (including but not limited to accelerometers, vibration sensors, etc.) to collect vibration signals of the power equipment and save them as vibration data files; next, use high-definition video equipment (including but not limited to fixed-position cameras or drones, etc.) to capture images of the working area of ​​the power equipment, record images of the power equipment covering key components and save them as image files (including but not limited to JPEG or PNG formats, etc.);

[0188] S12: Label the acquired audio files, vibration data files, and image files; determine whether the current device has a fault from the audio data in the audio files, and then label the audio data as either faulty or fault-free; determine whether the current device has a fault from the vibration signals in the vibration data files, and then label the vibration signals as either faulty or fault-free; determine whether the current device has a fault from the power equipment images in the image files, and then use the image labeling tool to label the power equipment images as either faulty or fault-free.

[0189] S1 Example:

[0190] Audio files, vibration data files, and image files were collected and labeled to obtain the original multimodal dataset of power equipment. The specific steps are as follows:

[0191] First, audio files were collected: audio data during the operation of the electrical equipment was collected using a microphone device, with the sampling frequency set to 16kHz and the recording duration set to 10 seconds; 2000 audio data points were collected and saved as audio files in WAV format.

[0192] Secondly, vibration data files were collected: accelerometers were used to collect vibration signals from the power equipment at a sampling frequency of 100Hz for 10 seconds; 2000 sets of vibration signals were collected and saved as vibration data files in TXT format, with each file containing 1000 data points.

[0193] Then, image files were collected: the working area of ​​the power equipment was photographed using high-definition camera equipment (such as a fixed camera or drone) to obtain images of the power equipment; each power equipment image file was 1920×1080 in size, a total of 1000 images were collected and saved as power equipment image files in JPEG format;

[0194] Finally, annotation was performed: audio data was manually annotated into two categories, "faulty" and "no fault," resulting in 1000 "faulty" data points and 1000 "no fault" data points; vibration signals were also manually annotated into two categories, "faulty" and "no fault," resulting in 1000 "faulty" data points and 1000 "no fault" data points; images of power equipment were annotated using an image annotation tool, classifying each image into two categories, "faulty" and "no fault," resulting in 500 "faulty" images and 500 "no fault" images.

[0195] S2: Preprocess the original power equipment multimodal dataset to obtain the preprocessed power equipment multimodal dataset;

[0196] The specific steps of S2 are as follows:

[0197] S21: Normalize the audio data in the audio file, that is, unify the sampling frequency and sampling duration, so that the audio data in the audio file has the same sampling interval, and obtain a one-dimensional sound subset.

[0198] S22: The audio data in the audio file is processed into a Mel spectrogram using Mel frequency cepstral coefficients to obtain a two-dimensional sound subset. The specific operation of the Mel frequency cepstral coefficients is as follows: First, the audio data is pre-emphasized to enhance high-frequency components and weaken the influence of low-frequency components; then, the audio data is segmented into frames; next, a Hamming window is used to window each frame of the audio signal in the audio data; then, a short-time Fourier transform (STFT) is performed on the windowed audio signal to obtain a spectrogram; then, the spectrogram is passed through a Mel filter bank to convert it into Mel frequency features; then, a logarithmic operation is performed on the Mel frequency features to obtain the Mel spectrum; finally, a discrete cosine transform (DCT) is performed on the Mel spectrum to extract the Mel spectrogram.

[0199] Mel-Frequency Cepstral Coefficients (MFCC) is an audio feature extraction method based on the characteristics of human hearing and is widely used in fields such as speech recognition. Its core idea is to use the Mel-frequency scale to simulate the nonlinear perception characteristics of human ears for sounds of different frequencies, because human ears are more sensitive to changes in low-frequency sounds than high-frequency sounds. MFCC can effectively capture important features in audio signals, such as formants and fundamental frequency information of speech, while ignoring some details that are not important for recognition.

[0200] Mel spectrogram is a time-frequency representation method based on the Mel scale. It converts the spectrum of a signal into a non-linear scale that conforms to the characteristics of human hearing, thus more effectively characterizing the perceptual features of sound. Unlike ordinary spectrograms, Mel spectrograms use Mel filter banks to non-uniformly divide the spectrum, with higher resolution in the low-frequency part and lower resolution in the high-frequency part, which is closer to the non-linear perception of pitch by the human auditory system. This representation method can highlight key frequency band information (such as formants and timbre) in signals such as speech, while reducing high-frequency redundancy.

[0201] The Hamming window is a commonly used window function in signal processing, primarily used to reduce spectral leakage during frequency domain analysis. When performing a Discrete Fourier Transform (DFT) on a signal, if the signal is not exactly periodic, its spectrum will exhibit undesirable sidelobes, leading to spectral leakage. The Hamming window effectively reduces this leakage by applying smooth weights to both ends of the signal, causing it to gradually decay to zero at both ends. It is a cosine window, shaped like a bell curve that is high in the middle and low at both ends, effectively balancing the relationship between the main lobe width and side lobe attenuation. In practical applications, the Hamming window is frequently used in speech signal processing, audio analysis, and radar signal processing to improve the accuracy and reliability of spectral analysis, making the spectrum clearer and facilitating the extraction of signal feature information.

[0202] S23: Normalize the vibration signals in the vibration data file to make their amplitudes within a uniform range, and then use the sliding window method to divide the vibration data into multiple time periods, each containing vibration data of uniform length, to obtain a one-dimensional vibration subset.

[0203] S24: Convert the vibration signals in the vibration data file into a Gram angular field map to obtain a two-dimensional vibration subset; first, normalize the vibration signals; then, map the amplitude and timestamp of each time point to the radius and angle of the polar coordinate system; subsequently, generate the Gram matrix by calculating the triangular inner product of all time point pairs; finally, perform trigonometric function mapping on the Gram matrix to obtain the Gram angular field map.

[0204] S25: Adjust the power equipment images in the image file to a uniform size, and then use data augmentation techniques (including but not limited to rotation, translation, cropping, flipping, adjusting brightness, contrast, etc.) to generate multiple enhanced images and add them to the dataset to obtain an image subset;

[0205] S26: Perform time alignment: For each sampling time point, extract the corresponding data samples from all subsets, including normalized audio data from the one-dimensional sound subset, Mel spectrograms from the two-dimensional sound subset, normalized vibration signals from the one-dimensional vibration subset, Gram angular field maps from the two-dimensional vibration subset, and images of power equipment from the image subset, and combine them to form a quintuple sample; the process achieves cross-modal synchronization through timestamp matching to ensure that the data samples of all subsets are aligned in the time dimension; repeat this alignment operation for all available time points to obtain the preprocessed multimodal dataset of power equipment;

[0206] S2 Implementation Example:

[0207] The original power equipment multimodal dataset is preprocessed to obtain the preprocessed power equipment multimodal dataset. The specific steps are as follows:

[0208] First, the audio data from 2000 audio files were normalized, with a uniform sampling frequency of 16kHz and a sampling duration of 10 seconds; to ensure that the sampling interval of the audio data was consistent, the length was 160,000 (16kHz × 10 seconds), resulting in a one-dimensional sound subset dataset.

[0209] Secondly, Mel frequency cepstral coefficient processing is performed on the audio data in the audio file to obtain a two-dimensional sound subset. The specific steps are as follows: First, the audio data is pre-emphasized with a filter coefficient of 0.97 to enhance the high-frequency part; Second, the audio data is divided into 25-millisecond frames with a 50% overlap (i.e., 160 samples per frame, with 80 overlapping samples); Third, a Hamming window is applied to each frame of audio data to obtain the windowed frame signal; Fourth, a short-time Fourier transform is performed on each windowed frame signal to obtain the spectrogram; Fifth, 40 Mel filters are used to map the spectrogram to the Mel frequency domain; Sixth, logarithmic operation is performed on the Mel frequency domain to obtain the Mel spectrum; Seventh, a discrete cosine transform is performed on the Mel spectrum to extract the Mel spectrogram; Eighth, the resulting Mel spectrograms, with a size of 40×2000 (40 Mel filters, 2000 frames), are combined to form a two-dimensional sound subset.

[0210] Then, the vibration signals in the 2000 vibration data files were normalized so that the data amplitude was uniformly between 0 and 1. Then, the normalized vibration signals were divided into multiple time periods by using the sliding window method (window length of 500 points, step size of 50 points), with each time period having a length of 500 data points, resulting in 80,000 vibration data time periods and obtaining a one-dimensional vibration subset.

[0211] Next, the vibration signals in the vibration data file are transformed to obtain a two-dimensional vibration subset. The specific steps are as follows: First, the vibration signals are normalized; second, the amplitude and timestamp of each time point are mapped to the radius and angle of the polar coordinate system; third, the triangular inner product of all time point pairs is calculated to generate a Gram matrix; fourth, the Gram matrix is ​​mapped using trigonometric functions to obtain a Gram angle field map; fifth, the obtained Gram angle field maps of size 100×100 (100×100 Gram matrix) are combined to form a two-dimensional vibration subset.

[0212] Next, the power equipment images in the 1000 image files were uniformly adjusted to a size of 224×224; then, data augmentation was performed on the images using operations such as rotation (maximum angle 30°), translation (maximum translation 20 pixels), cropping (maximum cropping ratio 0.2), flipping (horizontal flipping), and adjusting brightness and contrast to generate multiple augmented images and add them to the dataset, resulting in an image subset containing 2000 power equipment images;

[0213] Next, a rigorous time alignment operation is performed: For each sampling time point, data samples corresponding to that time are extracted from all subsets, including normalized audio data from the one-dimensional sound subset, Mel spectrograms from the two-dimensional sound subset, normalized vibration signals from the one-dimensional vibration subset, Gram angular field maps from the two-dimensional vibration subset, and images of power equipment from the image subset, combined to form a complete quintuple sample; this process achieves cross-modal synchronization through timestamp matching, ensuring that the data samples of all subsets are accurately aligned in the time dimension; this alignment operation is repeated for all available time points to obtain the preprocessed multimodal dataset of power equipment;

[0214] S3: Construct an anomaly detection model for power equipment based on multimodal deep learning;

[0215] This invention designs a power equipment anomaly detection model based on multimodal deep learning, which specifically includes the processing, feature extraction, and decision-level fusion of three modal data: audio data, vibration signals, and power equipment images, and finally outputs whether the power equipment has a fault.

[0216] The model consists of three sub-models and a multimodal weighted fusion algorithm: Sub-model M1 is used to detect power equipment anomalies in audio data from audio files. After processing the audio data through time-frequency co-feature fusion and classification, it outputs the predicted probability of power equipment failure and the predicted probability of failure in audio data. Sub-model M2 is used to detect power equipment anomalies in vibration signals from vibration data files. After processing the vibration signals through time-series joint feature fusion and classification, it outputs the predicted probability of power equipment failure and the predicted probability of failure in vibration data. Sub-model M3 is used to detect power equipment anomalies in images from image files. It processes the power equipment images through a network model and outputs the predicted probability of power equipment failure and the predicted probability of failure in image data. The multimodal weighted fusion algorithm performs decision-level fusion of the predicted probabilities of each sub-model and outputs the probability of whether the power equipment has a fault (divided into two categories: overall predicted probability of failure and overall predicted probability of failure), realizing a comprehensive assessment of the operating status of the power equipment. The overall architecture of the power equipment anomaly detection model is as follows: Figure 2 As shown;

[0217] The specific steps of S3 are as follows:

[0218] S31, Anomaly Detection Based on Power Equipment Audio Data: First, the Mel spectrogram from the two-dimensional sound subset of S2 is used to extract the frequency domain features of the audio data using MSFE-Net; then, the normalized audio data from the one-dimensional sound subset of S2 is used to extract the temporal domain features of the audio data using BiLSTM; subsequently, the results of the two are fused using time-frequency co-feature at the feature layer to obtain the audio feature map of the power equipment audio data; then, the audio feature map of the power equipment audio data is classified using an MLP classifier; finally, the classification results are converted into audio prediction of fault probability and audio prediction of no fault probability using a Softmax activation function.

[0219] Among them, the MLP (Multilayer Perceptron) classifier is a classic feedforward artificial neural network, consisting of an input layer, multiple hidden layers, and an output layer. It achieves non-linear classification through a fully connected structure. It utilizes the backpropagation algorithm to optimize network weights and introduces non-linear transformations using activation functions such as ReLU and Sigmoid. It can automatically learn hierarchical feature representations of input data. MLP enhances model capacity by stacking hidden layers, and the final output layer (usually in conjunction with the Softmax activation function) generates the class probability distribution. It is suitable for structured data classification tasks, but for unstructured data such as raw images and time-series signals, it needs to be used in conjunction with feature engineering. Its advantages are its simplicity and versatility.

[0220] The specific steps of S31 are as follows:

[0221] S311: Extract the frequency domain features of the audio data from the Mel spectrograms in the two-dimensional sound subset using the Mel spectrogram feature extraction network MSFE-Net, and obtain the Mel high-level feature map F. 19 ;

[0222] This invention designs a Mel-Spectrogram Feature Extraction Network (MSFE-Net), which is divided into five progressively deeper layers. The first layer initially extracts low-level features from the Mel spectrogram, capturing basic patterns and frequency components. The second layer further extracts more complex low-level features, identifying finer frequency patterns and local variations in the Mel spectrogram, revealing short-term spectral changes. The third layer begins extracting intermediate features, identifying complex structural patterns in the Mel spectrogram and focusing on higher-dimensional spectral information. These intermediate features contain more information than low-level features, resulting in a lower resolution but an increased number of channels in the feature map. The fourth layer extracts higher-level features, primarily identifying global patterns, long-term frequency and time dependencies, and potential changes in device state in the Mel spectrogram. The resulting feature map resolution is further reduced, but the number of channels remains high. The fifth layer extracts the most abstract features, which highly integrate feature information from all previous layers, primarily used to capture the overall behavior patterns, long-term trends, and potential anomalies of the device. The overall structure of MSFE-Net is as follows: Figure 3 As shown;

[0223] The specific steps of S311 are as follows:

[0224] S3111: Represent any Mel spectrogram from the two-dimensional sound subset as F1, input it into a convolutional layer with 64 3×3 kernels and a stride of 1, and then sequentially pass it through the ReLU activation function and batch normalization (i.e., ... Figure 3 The BN operation (Batch Normalization) is used to obtain the Mel shallow feature map F2. Then, F2 is input into a convolutional layer with 64 3×3 kernels and a stride of 1. Subsequently, it is processed by ReLU activation function and batch normalization to obtain the Mel shallow feature map F3.

[0225] S3112: Input F3 into the SAPM module to obtain the Mel shallow feature map F4;

[0226] This invention designs a Spatial Adaptive Pooling module (SAPM), the overall structure of which is as follows: Figure 4 As shown;

[0227] The shallow feature map F3 of the Mel layer is used as the input feature map of the SAPM module; for ease of description, F3 is denoted as A1 here;

[0228] Input a shallow Mel-ray feature map A1 of size H×W×C, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels in the feature map; after passing A1 through the SAPM module, obtain a shallow Mel-ray feature map A2 of size H / 2×W / 2×C. The specific operation is as follows: First, split A1 according to the channel dimension (i.e., Figure 4 The channel dimension splitting operation (ChanSplit) in the middle yields C independent single-channel feature maps a. 1_1 A 1_p A 1_z Then each single-channel feature map a 1_p (p∈[1,z]), each channel is processed by the single-channel spatial adaptive pooling module SSAPM to obtain the single-channel feature map A with the same channel number. 2_p Finally, the pooled C single-channel feature maps A 2_1 A 2_p A 2_z Reorganized according to channel dimension (i.e.) Figure 4 The channel dimension reorganization operation (ChanMerge) in the middle is used to obtain A2;

[0229] The following uses the p-th branch as an example to illustrate how to transform the single-channel feature map a 1_p After passing through the SSAPM module, a single-channel feature map A with a size of H / 2×W / 2 is obtained. 2_p The operation process of the p-th branch is the same as that of the p-th branch; the overall structure of SSAPM is as follows: Figure 5 As shown; Input is a single-channel feature map A of size H×W. 1_p To the seven branches of the SSAPM module;

[0230] The specific steps of S3112 are as follows:

[0231] S31121: In the first branch, calculate A along the vertical axis. 1_p The mean (i.e.) Figure 5 The vertical mean vector is calculated using the Average_H operation, resulting in the vertical mean vector μ. p,h In the second branch, calculate A along the vertical axis. 1_p variance (i.e.) Figure 5 The Variance_H operation in the middle (the part of the text) is used to obtain the vertical variance vector.

[0232] μp,h =[μ p,h [1],μ p,h [2],…,μ p,h [h]

[0233]

[0234] In the above formula, A 1_p (i,j) represents the state in A. 1_p The eigenvalue at position (i,j) in the middle;

[0235] S31122: In the third branch, according to A 1_p Calculate the height H to output the height H 2_p (Right now Figure 5 The operation to calculate the output height (Calculate_H) is performed within the context of this process.

[0236]

[0237] S31123: In the fourth branch, A is not correct. 1_p Perform any operation; in the fifth branch, calculate A along the horizontal axis. 1_p The mean (i.e.) Figure 5 The operation Average_W in the middle (of the text) yields the average water vector μ. p,w In the sixth branch, calculate A along the horizontal axis. 1_p variance (i.e.) Figure 5 The operation Variance_W in the middle class calculates the level variance vector.

[0238] μ p,w =[μ p,w [1],μ p,w [2],…,μ p,w [w]

[0239]

[0240]

[0241] In the above formula, A 1_p (i,j) represents the state in A. 1_p The eigenvalue at position (i,j) in the middle;

[0242] S31124: In the seventh branch, according to A 1_p The width W is used to calculate the output width W. 2_p (Right now Figure 5 The operation to calculate the output width (Calculate_W) is performed within the `<output_width>` function.

[0243]

[0244] S31125: Based on variance and Calculations are performed to obtain the variance factor. Then according to μ p,h Variance-guided fusion calculation was performed to obtain the mean μ. h Then according to Will Variance-guided fusion calculation is performed to obtain the variance. (Right now Figure 5 The variance-guided fusion operation (VarFusion) in the middle; then according to μ p,w Variance-guided fusion calculation was performed to obtain the mean μ. w Then according to Will Variance-guided fusion calculation was performed to obtain the mean. (Right now Figure 5 The variance-guided fusion operation VarFusion is used in the process; subsequently, based on μ... h and The calculation is performed to obtain the vertical pooling window size k. h According to μ w and The calculation is performed to obtain the horizontal pooling window size k. w ;

[0245]

[0246] In the above formula, max() represents the operation of calculating the maximum value; mean() represents the operation of calculating the average value; round() represents the operation of rounding the number to the nearest integer; k min ε is the size of the minimum pooling window, used to avoid the pooling window being too small; ε is a constant used to prevent instability caused by the mean being close to zero or the variance being too small.

[0247] S31126: According to k h and H 2_p Calculate the vertical step size s h Then, according to k w and W 2_p Calculate the horizontal step size s w ;

[0248]

[0249] In the above formula, round() means rounding the number to the nearest integer.

[0250] S31127: According to A1_p Height H, Output Height H 2_p Vertical pooling window size k h and vertical step size s h Calculate the vertical padding amount. h According to A 1_p Width W, Output Width W 2_p Horizontal pooling window size k w and horizontal step size s w Calculate the horizontal padding amount. w ;

[0251]

[0252] Next, for A 1_p Perform zero-fill on all four sides sequentially: first in A 1_p Add padding to the left w Fill the column with zero, then add padding on the right. w Fill the column with zero, then add padding at the top. h Use zero padding in the rows, then add padding at the bottom. h Row zero padding; ultimately resulting in a size of (H + 2 * padding) h )×(W+2*padding w The feature map of the four sides with zero fill;

[0253] Finally, based on the calculated pooling window size k h k w and step size s h s w Perform max pooling on the feature map with zero padding on all four sides (i.e., Figure 5 The max pooling operation (MaxPool) in the model yields a single-channel feature map A of size H / 2 × W / 2. 2_p ;

[0254] Example:

[0255] Input a shallow Mel-ray feature map A1 of size 224×224×64 (where the height H of the feature map is 224, the width W is 224, and the number of channels C is 64); after passing A1 through the SAPM module, a shallow Mel-ray feature map A2 of size 112×112×64 is obtained. Specifically, A1 is first split according to the channel dimension to obtain 64 independent single-channel feature maps a. 1_1 A 1_p A 1_64 Then each single-channel feature map A 1_p(p∈[1,64]) respectively pass through the single-channel spatial adaptive pooling module SSAPM to obtain the single-channel feature map A of the same channel number. 2_p (p∈[1,64]); Finally, the 64 pooled single-channel feature maps A 2_1 A 2_p A 2_64 Reorganize according to the channel dimension to obtain A2;

[0256] Taking the p-th branch (p∈[1,64]) as an example, this explains how to transform the single-channel feature map A... 1_p After passing through the SSAPM module, a single-channel feature map A with the same channel number is obtained. 2_p Operation process:

[0257] Input a single-channel feature map A of size 224×224 1_p To the seven branches of the SSAPM module;

[0258] In the first branch, calculate A along the vertical axis. 1_p The mean value is used to obtain the vertical mean vector μ. p,h ;

[0259] μ p,h = [0.11,…,0.09,…,0.3]

[0260] In the second branch, a is calculated along the vertical axis. 1_p The variance is used to obtain the vertical variance vector.

[0261]

[0262] In the third branch, according to A 1_p Calculate the height H to output the height H 2_p ;

[0263]

[0264] In the fourth branch, A is not correct. 1_p Perform any operation;

[0265] In the fifth branch, A is calculated along the horizontal axis. 1_p The mean value is used to obtain the water average vector μ. p,w ;

[0266] μ p,w = [0.09,…,0.3,…,0.14]

[0267] In the sixth branch, calculate A along the horizontal axis. 1_p The variance is used to obtain the level variance vector.

[0268]

[0269] In the seventh branch, according to A 1_p The width W is used to calculate the output width W. 2_p ;

[0270]

[0271] According to variance and Calculations are performed to obtain the variance factor.

[0272]

[0273] according to μ p,h Variance-guided fusion calculation was performed to obtain the mean μ. h ;

[0274]

[0275] according to Will Variance-guided fusion calculation is performed to obtain the variance.

[0276]

[0277] according to μ p,w Variance-guided fusion calculation was performed to obtain the mean μ. w ;

[0278]

[0279] according to Will Variance-guided fusion calculation was performed to obtain the mean.

[0280]

[0281] According to μ h and The calculation is performed to obtain the vertical pooling window size k. h ;

[0282]

[0283] According to μ w and The calculation is performed to obtain the horizontal pooling window size k. w ;

[0284]

[0285] In the above formula, k min Take 2; take 10. -6 ;

[0286] According to k h and H 2_p Calculate the vertical step size s h ;

[0287]

[0288] According to k w and W 2_p Calculate the horizontal step size s w ;

[0289]

[0290] According to A 1_p Height H, Output Height H 2_p Vertical pooling window size k h and vertical step size s h Calculate the vertical padding amount. h ;

[0291]

[0292] According to A 1_p Width W, Output Width W 2_p Horizontal pooling window size k w and horizontal step size s w Calculate the horizontal padding amount. w ;

[0293]

[0294] Next, for A 1_p Perform zero-fill on all four sides sequentially: first in A 1_p Add one column of zero-fill to the left, then add one column of zero-fill to the right, then add two rows of zero-fill to the top, and then add two rows of zero-fill to the bottom; finally, you get a feature map with four zero-fill sides of size 228×226.

[0295] Finally, based on the calculated vertical pooling window size k h =5. Horizontal pooling window size k w =4 and vertical step size s h =2, horizontal step size s w =2, perform max pooling on the feature map with zero padding on all four sides to obtain a single-channel feature map A of size 112×112. 2_p ;

[0296] S3113: Input F4 into a convolutional layer with 128 3×3 kernels and a stride of 1, then pass it through the ReLU activation function and batch normalization to obtain the Mel shallow feature map F5; then input F5 into a convolutional layer with 128 3×3 kernels and a stride of 1, then pass it through the ReLU activation function and batch normalization to obtain the Mel shallow feature map F6; finally, input F6 into the SAPM module to obtain the Mel shallow feature map F7;

[0297] S3114: Input F7 into a convolutional layer with 256 3×3 kernels and a stride of 1, then pass it through ReLU activation and batch normalization to obtain the Mel-Ray mid-layer feature map F8; then input F8 into a convolutional layer with 256 3×3 kernels and a stride of 1, then pass it through ReLU activation and batch normalization to obtain the Mel-Ray mid-layer feature map F9; finally, input F9 into the SAPM module to obtain the Mel-Ray mid-layer feature map F... 10 ;

[0298] S3115: F 10 The input is fed into a convolutional layer with 512 3×3 kernels and a stride of 1, and then sequentially processed by ReLU activation and batch normalization to obtain the Mel high-level feature map F. 11 Then F 11 The input is fed into a convolutional layer with 512 3×3 kernels and a stride of 1, and then sequentially processed by ReLU activation and batch normalization to obtain the Mel high-level feature map F. 12 ;

[0299] S3116: F 12 The input is fed into the LGFA module to obtain the Mel high-level feature map F. 13 ;

[0300] This invention designs a Local-Global FusionAttention module (LGFA), the overall structure of which is as follows: Figure 6 As shown;

[0301] Meyer's high-level feature map F 13 The feature map is used as the input to the LGFA module. For ease of explanation, F will be referred to here. 13 Noted as B1;

[0302] Input a Mel high-level feature map B1 of size H×W×C into five branches, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map;

[0303] The specific steps of S3116 are as follows:

[0304] S31161: In the first branch, no operation is performed on B1; in the second branch, global average pooling is first performed on B1 (i.e., Figure 6 The GAP operation in the middle is used to obtain the local attention feature map B2; a fully connected layer (i.e., Figure 6 The FC1 module in the code performs dimensionality reduction on B2 to obtain the local attention feature map B3; then B3 is processed through the ReLU activation function to obtain the local attention feature map B4; finally, another fully connected layer (i.e., ...) is used to perform dimensionality reduction on B2. Figure 6 The FC2 module in the code performs a dimensionality increase operation on B4 to obtain the local attention feature map B5, making the number of channels in B5 the same as that in B2; then, B5 is passed through the Sigmoid activation function to obtain the local attention feature map B6; finally, B6 is multiplied element-wise with B1 (i.e., ...). Figure 6 The element-wise multiplication operation (⊙) is used to obtain the local attention feature map B7; then, global average pooling (i.e., ...) is performed on B7. Figure 6 The GAP operation in the middle), and then the feature transformation of the pooling result is performed by a multilayer perceptron (i.e., Figure 6 The MLP module in the code transforms the feature map by passing the result through the Sigmoid activation function to obtain the local attention feature map B8; finally, element-wise multiplication is performed between B7 and B8 (i.e., ...). Figure 6 The element-wise multiplication operation ⊙) is used to obtain the local attention feature map B9;

[0305] B7=B1⊙S(FC2(ReLU(FC1(GAP(B1)))))

[0306] B9 = B7⊙S(MLP(GAP(B7)))

[0307] In the above formula, S represents the Sigmoid activation function; GAP() represents the Global Avg Pool operation; ReLU represents the ReLU activation function; MLP() represents multilayer perceptron processing; FC1() represents dimensionality reduction processing through a fully connected layer; FC2() represents dimensionality increase processing through a fully connected layer; ⊙ represents element-wise multiplication operation.

[0308] Global Average Pooling (GAP) is a commonly used feature dimensionality reduction operation. Its core idea is to calculate the average value of all spatial locations (height × width) of each channel of the feature map, thereby compressing a two-dimensional feature map of arbitrary size into a one-dimensional vector. Specifically, for a feature map with an input size of H×W×C, GAP calculates the average value of H×W pixels channel by channel, and finally outputs a one-dimensional vector of length C, where each element corresponds to the global response intensity of a channel.

[0309] S31162: In the third branch, the weight matrix W v Multiply by B1 (i.e.) Figure 6 The matrix multiplication operation (×) is used to obtain the value matrix V; in the fourth branch, the weight matrix W is... q Multiply by B1 (i.e.) Figure 6 The matrix multiplication operation (×) is used to obtain the query matrix Q; in the fifth branch, the weight matrix W is... k Multiply by B1 (i.e.) Figure 6 The matrix multiplication operation (×) is used to obtain the key matrix K; then, the dot product of Q and K is calculated, and the result is activated by the Softmax activation function (i.e., ...). Figure 6 The attention score matrix is ​​calculated using the QKS operation, resulting in the attention score matrix S; then S is multiplied by V (i.e., ...). Figure 6 The matrix multiplication operation (×) in the matrix yields the global attention feature map B. 10 Next, a 1×1 convolution kernel is applied to B. 10 Perform convolution operations (i.e.) Figure 6 The convolution operation Conv1×1 in the middle yields the global attention feature map B. 11 Then, for B 11 Perform global average pooling (i.e.) Figure 6 The GAP operation in the middle), and then the feature transformation of the pooling result is performed by a multilayer perceptron (i.e., Figure 6 The MLP module in the code transforms the feature map by passing the result through a Sigmoid activation function to obtain the global attention feature map B. 12 Subsequently, B 11 With B 12 Perform element-wise multiplication (i.e.) Figure 6 The element-wise multiplication operation (⊙) in the algorithm yields the global attention feature map B. 13 ;

[0310] V = W v ×B1

[0311] Q = W q ×B1

[0312] K = W k ×B1

[0313]

[0314] B 12 =S(MLP(GAP(B) 11 )))

[0315] B 13 =B 11 ⊙B 12

[0316] In the above formula, S represents the Sigmoid activation function; Softmax represents the Softmax activation function; GAP() represents the Global Avg Pool operation; MLP() represents the multilayer perceptron processing; and the He initialization method is used to initialize the weight matrix W. q W k and W v Perform initialization, W q W k and W v All dimensions are Where C' is the number of channels after compression; × represents matrix multiplication; ⊙ represents element-wise multiplication; + represents element-wise addition;

[0317] S31163: Finally, connect B9 and B 13 Perform element-wise addition (i.e.) Figure 6 The element-wise addition operation (+) is used to obtain and output the fused attention feature map B. 14 ;

[0318] Among them, He initialization is a weight initialization method. Its core idea is to ensure that the input and output of each layer maintain similar variance, thereby avoiding the gradient vanishing or exploding problem. The steps for generating weights according to the He initialization formula are as follows:

[0319] The first step is to calculate the standard deviation:

[0320]

[0321] The second step is to generate the weight matrix: each element is randomly sampled from a normal distribution N(0,σ), resulting in W. q W k and W v ;

[0322] Furthermore, the subsequent changes in the weights are as follows: weight matrix W q W k and W v The value is updated through backpropagation and optimization algorithms (such as SGD, Adam), and the update follows the rules of the optimization algorithm:

[0323]

[0324] In the above formula, η is the learning rate. and This represents the gradient of the loss function with respect to the weight matrix.

[0325] Example:

[0326] The input is a 28×28×512 Mel high-level feature map B1, which is fed into five branches. In the first branch, no operation is performed on B1. In the second branch, B1 is first subjected to global average pooling to obtain a local attention feature map B2, which has a size of 1×1×512. A fully connected layer is used to reduce the dimensionality of B2 to obtain a local attention feature map B3, which has a size of 1×1×128. Then, B3 is passed through a ReLU activation function to obtain a local attention feature map B4, which has a size of 1×1×128. Finally, another fully connected layer is used to increase the dimensionality of B4 to obtain a local attention feature map B5, which has a size of 1×1. ×512; Next, B5 is activated by the Sigmoid function to obtain the local attention feature map B6, which has a size of 1×1×512; then, B6 is multiplied element-wise with B1 to obtain the local attention feature map B7, which has a size of 28×28×512; then, global average pooling is performed on B7; then, a multilayer perceptron is used to transform the pooling result, and the transformed result is activated by the Sigmoid function to obtain the local attention feature map B8, which has a size of 1×1×512; finally, B7 and B8 are multiplied element-wise to obtain the local attention feature map B9, which has a size of 28×28×512;

[0327] In the third branch, the weight matrix W v Multiplying by B1 yields the value matrix V, which has a size of 28×28×128; in the fourth branch, the weight matrix W is... q Multiplying by B1 yields the query matrix Q, which has a size of 28×28×128; in the fifth branch, the weight matrix W is... k Multiplying Q by B1 yields the key matrix K, which is 28×28×128 in size. Then, the dot product of Q and K is calculated and activated using the Softmax activation function to obtain the attention score matrix S, which is also 28×28×28 in size. Finally, S is multiplied by V to obtain the global attention feature map B. 10 B 10 The size is 28×28×128; then a 1×1 convolution kernel is applied to B. 10 Perform a convolution operation to obtain the global attention feature map B. 11 B 11 The size is 28×28×512; then, for B 11 Global average pooling is performed, followed by feature transformation of the pooling result using a multilayer perceptron. The transformed feature result is then passed through a sigmoid activation function to obtain the global attention feature map B. 12 B 12 The size is 1×1×512; then, B 11 With B 12Perform element-wise multiplication to obtain the global attention feature map B. 13 B 13 Its size is 28×28×512;

[0328] Finally, B9 and B 13 Perform element-wise addition to obtain and output the fused attention feature map B. 14 B 14 Its size is 28×28×512;

[0329] S3117: F 13 The input is fed into the SAPM module to obtain the Mel high-level feature map F. 14 Next, F 14 The input is fed into a convolutional layer with 512 3×3 kernels and a stride of 1, and then sequentially processed by ReLU activation and batch normalization to obtain the Mel high-level feature map F. 15 Then, F 15 The input is fed into a convolutional layer with 512 3×3 kernels and a stride of 1, and then sequentially processed by ReLU activation and batch normalization to obtain the Mel high-level feature map F. 16 After that, F 16 The input is fed into the LGFA module to obtain the Mel high-level feature map F. 17 Next, F 17 The input is fed into the SAPM module to obtain the Mel high-level feature map F. 18 Finally, F 18 The input is fed into a fully connected layer, and the high-level feature map F of Mel is obtained and output. 19 ;

[0330] Example S31:

[0331] The Mel spectrogram F1 from the two-dimensional sound subset obtained in S2 is used as the input to the Mel spectrogram feature extraction network. F1 has a size of 224×224 pixels and 1 channel. F1 is then fed into a convolutional layer with 64 3×3 kernels and a stride of 1, followed by a ReLU activation function and a BN layer to obtain the Mel shallow feature map F2, which has a size of 224×224 pixels and 64 channels. F2 is then fed into another convolutional layer with 64 3×3 kernels and a stride of 1, followed by a ReLU activation function and a BN layer to obtain the Mel shallow feature map F3, which has a size of 224×224 pixels and 64 channels. F3 is then fed into the SAPM module to obtain the Mel shallow feature map F4, which has a size of 112×112 pixels and 64 channels.

[0332] F4 is input into a convolutional layer with 128 3×3 kernels and a stride of 1, followed by a ReLU activation function and a BN layer to obtain the Mel shallow feature map F5, which is 112×112 pixels in size and has 128 channels. F5 is then input into a convolutional layer with 128 3×3 kernels and a stride of 1, followed by a ReLU activation function and a BN layer to obtain the Mel shallow feature map F6, which is 112×112 pixels in size and has 128 channels. Finally, F6 is input into the SAPM module to obtain the Mel shallow feature map F7, which is 56×56 pixels in size and has 128 channels.

[0333] F7 is input into a convolutional layer with 256 3×3 kernels and a stride of 1, followed by a ReLU activation function and a BN layer to obtain the Mel mid-layer feature map F8, which is 56×56 pixels in size and has 256 channels. F8 is then input into another convolutional layer with 256 3×3 kernels and a stride of 1, followed by a ReLU activation function and a BN layer to obtain the Mel mid-layer feature map F9, which is also 56×56 pixels in size and has 256 channels. Finally, F9 is input into the SAPM module to obtain the Mel mid-layer feature map F... 10 F 10 The size is 28×28 pixels, and the number of channels is 256;

[0334] F 10 The input is fed into a convolutional layer with 512 3×3 kernels and a stride of 1, followed by a ReLU activation function and a BN layer to obtain the Mel high-level feature map F. 11 F 11 The size is 28×28 pixels, and the number of channels is 512; then F 11 The input is fed into a convolutional layer with 512 3×3 kernels and a stride of 1, followed by a ReLU activation function and a BN layer to obtain the Mel high-level feature map F. 12 F 12 The size is 28×28 pixels, and the number of channels is 512; then, F 12 The input is fed into the LGFA module to obtain the Mel high-level feature map F. 13 F 13 The size is 28×28 pixels, and the number of channels is 512; F 13 The input is fed into the SAPM module to obtain the Mel high-level feature map F. 14 F 14 The size is 14×14 pixels, and the number of channels is 512;

[0335] F 14The input is fed into a convolutional layer with 512 3×3 kernels and a stride of 1, followed by a ReLU activation function and a BN layer to obtain the Mel high-level feature map F. 15 F 15 The size is 14×14 pixels, and the number of channels is 512; then F 15 The input is fed into a convolutional layer with 512 3×3 kernels and a stride of 1, followed by a ReLU activation function and a BN layer to obtain the Mel high-level feature map F. 16 F 16 The size is 14×14 pixels, and the number of channels is 512; then, F 16 The input is fed into the LGFA module to obtain the Mel high-level feature map F. 17 F 17 The size is 14×14 pixels, and the number of channels is 512; next, F 17 The input is fed into the SAPM module to obtain the Mel high-level feature map F. 18 F 18 The size is 7×7 pixels, and the number of channels is 512; finally, F 18 The input is fed into a fully connected layer, and the high-level feature map F of Mel is obtained and output. 19 F 19 The feature dimension is 512;

[0336] S312: Extracting Temporal Features from Audio Data: The normalized audio data from the one-dimensional sound subset is input into a Bidirectional Long Short-Term Memory (BiLSTM) network. First, the forward LSTM layer in the BiLSTM processes the audio data from left to right step by step, progressively updating the forward hidden state to capture the forward dependencies of the audio data in the time series. Simultaneously, the backward LSTM layer in the BiLSTM processes the audio data from right to left step by step, updating the backward hidden state to capture the backward dependencies of the audio data. Next, the forward and backward hidden states are concatenated at each time step to obtain the bidirectional features of that time step. Finally, global average pooling is performed on the bidirectional features of all time steps to obtain the temporal feature map E1.

[0337] Among them, Bidirectional Long Short-Term Memory (BiLSTM) is a recurrent neural network structure that combines forward and backward LSTMs, capable of simultaneously capturing the forward and backward dependencies of time series data. Its core consists of two independent LSTM layers: the forward LSTM layer processes the input sequence in chronological order, learning the influence of historical information on the current state; the backward LSTM layer processes the sequence in reverse order, modeling the dependence of future information on the current state; the forward and backward hidden states at each time step are fused by concatenation or summation to form a feature representation containing complete contextual information; compared with unidirectional LSTM, BiLSTM significantly improves sequence modeling capabilities through bidirectional information flow, making it particularly suitable for tasks requiring global context awareness.

[0338] S313: F 19 The audio feature map of the power equipment audio data is obtained by concatenating E1 along the feature dimension. Then, an MLP classifier is used to classify the audio feature map of the power equipment audio data, resulting in a two-dimensional vector containing the audio fault logits score and the audio non-fault logits score. Finally, the two-dimensional vector is normalized using the Softmax activation function to obtain the audio fault prediction probability. And audio prediction fault-free probability

[0339] S32, Anomaly Detection Based on Vibration Signals of Power Equipment: First, the Gram angular field map from the two-dimensional vibration subset of S2 is used to extract the spatial features of the vibration signal using GAFE-Net; then, the normalized vibration signal from the one-dimensional vibration subset of S2 is used to extract the temporal features of the vibration signal using BiGRU; subsequently, the results of the two are fused at the feature layer to obtain the vibration feature map of the power equipment vibration signal; then, the vibration feature map of the power equipment vibration signal is classified using an MLP classifier; finally, the classification results are converted into the vibration prediction probability of fault and the vibration prediction probability of no fault using a Softmax activation function.

[0340] The specific steps of S32 are as follows:

[0341] S321: Extract the spatial features of the vibration signal from the Gram angular field map in the two-dimensional vibration subset using the Gram angular field map feature extraction network GAFE-Net, and obtain the Gram extracted feature map G. 15 ;

[0342] This invention designs a Gramian Angular Field Feature Extraction Network (GAFE-Net), the overall structure of which is as follows: Figure 7 As shown;

[0343] The specific steps of S321 are as follows:

[0344] S3211: Represent any Gram angular field map from the two-dimensional vibration subset as G1, input it into a convolutional layer with a kernel size of 4×4 and a stride of 4, and then perform layer normalization to obtain the Gram extracted feature map G2; then, pass G2 through 3 consecutive Bottleneck modules to obtain the Gram extracted feature map G3; then, pass G3 through a Downsample module to reduce its height by half, width by half, and keep the number of channels unchanged to obtain the Gram extracted feature map G4. Specifically, first, pass G3 through layer normalization, then pass the normalized result through a convolutional layer with a kernel size of 2×2 and a stride of 2 to obtain G4; then, pass G4 through 5 consecutive Bottleneck modules to obtain the Gram extracted feature map G5.

[0345] Layer Normalization (LN) is a technique used to stabilize and accelerate the training process of neural networks. Unlike batch normalization, layer normalization does not rely on the statistical information of small batches of data, but rather performs normalization operations on the features of individual samples. Specifically, for each sample, layer normalization calculates the mean and standard deviation of all features of that sample, and then normalizes each feature to have zero mean and unit variance. This normalization method can effectively reduce internal covariate bias and improve the training speed and stability of the model.

[0346] The Bottleneck module is a high-efficiency convolutional neural network architecture unit consisting of three convolutional layers: the first is a 1×1 convolutional layer for dimensionality reduction, compressing the number of channels in the input feature map to reduce subsequent computation; the second is a 3×3 convolutional layer responsible for feature extraction, with computational cost reduced accordingly since the number of input channels has been compressed; and the third is a 1×1 convolutional layer for dimensionality increase, restoring the number of channels to an appropriate size to match the subsequent network structure. This structure effectively reduces computational complexity by decreasing the number of channels in the intermediate feature maps, while preserving feature information well, making the network more efficient when processing large-scale data.

[0347] S3212: Input G5 into the DAACO module to obtain Gram extracted feature map G6;

[0348] This invention designs a Dynamic Anomaly-Aware Channel Optimization (DAACO) module, the overall structure of which is as follows: Figure 8 As shown;

[0349] The Gram-extracted feature map G5 is used as the input feature map for the DAACO module. For ease of explanation, G5 is denoted as L1 here. The Gram-extracted feature map L1 is input into four branches;

[0350] The specific steps of S3212 are as follows:

[0351] S32121: In the first branch, no operation is performed on L1; in the second branch, global average pooling is applied to L1 to obtain the channel-optimized feature map L2 (i.e., Figure 8 Global average pooling operation AvgPool 1×1 In the third branch, a pooling window of size 3×3 and a dynamic step size s are first applied. D Perform average pooling on L1, then perform global average pooling on the result to obtain the channel-optimized feature map L3 (i.e., Figure 8 The 3×3 two-stage pooling operation AvgPool 3×3 In the fourth branch, a pooling window of size 5×5 and a dynamic step size s are first applied. D Perform average pooling on L1, then perform global average pooling on the result to obtain the channel-optimized feature map L4 (i.e., Figure 8 The 5x5 two-stage pooling operation AvgPool 5×5 );

[0352]

[0353] L2 = GAP(L1)

[0354] L3 = GAP(AvgPool3(L1))

[0355] L4 = GAP(AvgPool5(L1))

[0356] In the above formula, It is a hyperparameter; max() indicates the maximum value calculation operation; GAP() indicates the Global AvgPool operation; AvgPool3() indicates a pooling window size of 3×3 and a stride of s. D Average pooling operation; AvgPool5() represents a pooling window size of 5×5 with a stride of s. D Average pooling operation;

[0357] S32122: Adjust the channel sparse parameter W S1 Multiply by L2 (i.e.) Figure 8 The parameters in the multiplication operation (×) are used to obtain the channel optimized feature map L5; the channel sparse parameters W are then multiplied.S2 Multiply by L3 (i.e.) Figure 8 The parameters in the multiplication operation (×) are used to obtain the channel optimized feature map L7; the channel sparse parameters W are then multiplied. S3 Multiply by L4 (i.e.) Figure 8 The parameters in the multiplication operation (×) are then used to obtain the channel optimized feature map L9; then L5, L7 and L9 are added element-wise (i.e. Figure 8 The element-wise addition operation (+) is used to obtain the channel optimized feature map L. 11 Then L 11 After applying the Sigmoid activation function, the channel-optimized feature map L is obtained. 12 Next, L 12 Perform element-wise multiplication with L1 (i.e.) Figure 8 The element-wise multiplication operation (⊙) in the middle yields the channel optimized feature map L. 13 ;

[0358] L 13 =S(W S1 ×L2+W S2 ×L3+W S3 ×L4)⊙L1

[0359] In the above formula, the Xavier initialization method is used to initialize the channel sparse parameter W. S1 W S2 and W S3 Initialize; × represents matrix multiplication; ⊙ represents element-wise multiplication; + represents element-wise addition;

[0360] S32123: Transfer abnormal channel parameter W A1 Multiply by L2 (i.e.) Figure 8 The parameters in the multiplication operation (×) are used to obtain the channel optimization feature map L6; the abnormal channel parameters W are then used to obtain the channel optimization feature map L6. A2 Multiply by L3 (i.e.) Figure 8 The parameters in the multiplication operation (×) are used to obtain the channel optimization feature map L8; the abnormal channel parameters W are then used to obtain the channel optimization feature map L8. A3 Multiplying by L4 yields the channel-optimized feature map L. 10 Then add L6, L8 and L 10 Perform element-wise addition (i.e.) Figure 8 The parameters in the multiplication operation (×) are used to obtain the channel optimized feature map L. 14 Then L 14 After applying the ReLU activation function, the channel-optimized feature map L is obtained. 15 ;

[0361] L 15 =ReLU(W A1 ×L6+W A2 ×L8+W A3 ×L10 )

[0362] The above formula uses a Gaussian distribution. The initialization method addresses the abnormal channel parameter W. A1 W A2 and W A3 Initialize; ReLU represents the ReLU activation function; × represents matrix multiplication; + represents element-wise addition.

[0363] S32124: L 13 With L 15 Perform element-wise multiplication (i.e.) Figure 8 The element-wise multiplication operation (⊙) in the middle yields the channel optimized feature map L. 16 Then, the channel attention parameter W D With L 16 Perform multiplication (i.e.) Figure 8 The parameters in the multiplication operation (×) are used to obtain the channel optimized feature map L. 17 Then L 17 After applying the Softmax activation function, the channel-optimized feature map L is obtained. 18 Finally, L 16 With L 18 Perform element-wise multiplication (i.e.) Figure 8 The element-wise multiplication operation (⊙) is used to obtain and output the channel optimized feature map L. 19 ;

[0364] L 19 =L 16 ⊙Softmax(W D ×(L 13 ⊙L 15 ))

[0365] The above formula uses a Gaussian distribution. Initialization method, for channel attention parameter W D Initialize; Softmax represents the Softmax activation function; × represents matrix multiplication; ⊙ represents element-wise multiplication.

[0366] Among them: (1) Xavier initialization

[0367] Xavier initialization keeps initial weight values ​​small to prevent gradient vanishing or exploding. It is suitable for networks using Sigmoid or Tanh activation functions, and the formula is as follows:

[0368]

[0369] In the above formula, n in It is the number of input channels, n out This refers to the number of output channels;

[0370] (2) Gaussian distribution initialization

[0371] Gaussian initialization is suitable for ReLU or LeakyReLU networks, maintaining stable gradient updates, where the formula is:

[0372]

[0373] In the above formula, the Gaussian distribution has a mean of 0 and a standard deviation of 0.02, indicating that the initial values ​​of the parameters are concentrated around 0, and ~ indicates that it follows a probability distribution;

[0374] Example:

[0375] The input is a 28×28×128 gram, from which feature map L1 is extracted and distributed into four branches. In the first branch, no operation is performed on L1. In the second branch, global average pooling is applied to L1 to obtain channel-optimized feature map L2, which has a size of 1×1×128. In the third branch, a pooling window of size 3×3 and a dynamic stride s are first applied. D =2. Perform average pooling on L1, and then perform global average pooling on the result to obtain channel-optimized feature map L3, which has a size of 1×1×128; In the fourth branch, a pooling window of size 5×5 and a dynamic step size s are first used. D =2. Perform average pooling on L1, and then perform global average pooling on the result to obtain the channel-optimized feature map L4, which has a size of 1×1×128; where the dynamic step size s D The calculation process is as follows:

[0376]

[0377] Subsequently, the channel sparse parameter W S1 Multiplying by L2 yields the channel-optimized feature map L5, which has a size of 1×1×128; the channel sparsity parameter W is then... S2 Multiplying by L3 yields the channel-optimized feature map L7, which has a size of 1×1×128; the channel sparsity parameter W is then... S3 Multiplying by L4 yields the channel-optimized feature map L9, which has a size of 1×1×128; then, element-wise addition of L5, L7, and L9 yields the channel-optimized feature map L... 11 L 11 The size is 1×1×128; then L 11 After applying the Sigmoid activation function, the channel-optimized feature map L is obtained. 12 L 12 The size is 1×1×128; then L12 Element-wise multiplication with L1 yields the channel-optimized feature map L. 13 L 13 The size is 28×28×128;

[0378] Secondly, the abnormal channel parameter W A1 Multiplying by L2 yields the channel-optimized feature map L6, which has a size of 1×1×128; the abnormal channel parameters W are then... A2 Multiplying by L3 yields the channel-optimized feature map L8, which has a size of 1×1×128; the abnormal channel parameters W are then... A3 Multiplying by L4 yields the channel-optimized feature map L. 10 L 10 The size is 1×1×128; then L6, L8 and L 10 Perform element-wise addition to obtain the channel-optimized feature map L. 14 L 14 The size is 1×1×128; then L 14 After applying the ReLU activation function, the channel-optimized feature map L is obtained. 15 L 15 The size is 1×1×128;

[0379] Next, first put L 13 With L 15 Perform element-wise multiplication to obtain the channel-optimized feature map L. 16 L 16 The size is 28×28×128; then, the channel attention parameter W... D With L 16 Perform a multiplication operation to obtain the channel-optimized feature map L. 17 L 17 The size is 28×28×128; then L 17 After applying the Softmax activation function, the channel-optimized feature map L is obtained. 18 L 18 The size is 28×28×128; finally, L 16 With L 18 Perform element-wise multiplication to obtain and output the channel-optimized feature map L. 19 L 19 The size is 28×28×128;

[0380] S3213: Input G6 into the Downsample module to reduce its height and width by half while keeping the number of channels unchanged, to obtain the Gram extraction feature map G7; then, pass G7 through 7 Bottleneck modules to obtain the Gram extraction feature map G8; then, pass G8 through the DAACO module to obtain the Gram extraction feature map G9.

[0381] S3214: Input G9 into the MLAF module to obtain the Gram extracted feature map G. 10 ;

[0382] This invention designs a multi-level adaptive fusion module (MLAF), the overall structure of which is as follows: Figure 9 As shown;

[0383] The Gram-extracted feature map G9 is used as the input feature map of the MLAF module; for ease of description, G9 is denoted as T1 here;

[0384] The input is a Gram of size H×W×C, from which feature map T1 is extracted into nine branches, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map.

[0385] The specific steps of S3214 are as follows:

[0386] S32141: In the first branch, no operation is performed on T1; in the second branch, no operation is performed on T1; in the third branch, wavelet transform is first performed on T1 to extract high-frequency anomalous features (i.e., Figure 9 The wavelet transform operation (DWT) is used to obtain a multi-level fused feature map T2; then, a convolution operation (i.e., a 1×1 convolution kernel) is applied to T2. Figure 9 The convolution operation (Conv1×1) is performed to obtain the multi-level fused feature map T3; then T3 is passed through the Sigmoid activation function to obtain the multi-level fused feature map T4.

[0387] T4 = S(Conv1(DWT(T1)))

[0388] In the above formula, DWT() represents the wavelet transform operation; Conv1() represents the convolution operation with a kernel of size 1×1; and S represents the sigmoid activation function.

[0389] Among them, Discrete Wavelet Transform (DWT) is a time-frequency analysis technique that decomposes a signal into approximate coefficients (low-frequency information) and detail coefficients (high-frequency information) at different scales to achieve multi-resolution analysis of the signal. Its core is to perform convolution and downsampling operations on the original signal using orthogonal wavelet basis functions (such as Haar, Daubechies, etc.), which effectively captures transient anomalies and local details while preserving the time-domain characteristics of the signal. Compared with Fourier Transform, wavelet Transform has adaptive window characteristics, making it suitable for processing non-stationary time-series data such as vibration signals of power equipment, and can accurately extract fault impact features and periodic components.

[0390] S32142: In the fourth branch, a depthwise separable convolution operation is performed on T1 using a 1×1 kernel (i.e., Figure 9 In the fifth branch, a depthwise separable convolution operation (DSC1×1) is performed on T1 to obtain a multi-level fused feature map T5; in the fifth branch, a depthwise separable convolution operation (DSC1×1) is performed on T1 using a 3×3 convolution kernel. Figure 9 In the sixth branch, a depthwise separable convolution operation (DSC3×3) is performed on T1 to obtain a multi-level fused feature map T6; in the sixth branch, a depthwise separable convolution operation (i.e., DSC3×3) is performed on T1 using a 5×5 convolution kernel. Figure 9 In the seventh branch, a depthwise separable convolution operation (DSC5×5) is performed on T1 to obtain a multi-level fused feature map T7; in the seventh branch, a depthwise separable convolution operation (i.e., DSC5×5) is performed on T1 using a 7×7 convolution kernel. Figure 9 The depthwise separable convolution operation (DSC7×7) is used to obtain a multi-level fused feature map T8.

[0391] Among them, Depthwise Separable Convolution is an efficient convolution operation that decomposes standard convolution into two independent steps to reduce computational cost. First, it performs channel-wise spatial convolution, where each input channel uses a separate two-dimensional convolution kernel to extract spatial features while keeping the number of channels constant. Then, it fuses cross-channel information through 1×1 pointwise convolution to adjust the output channel dimension. This structure can significantly reduce the number of parameters and computational cost while maintaining good feature extraction capabilities.

[0392] Next, perform element-wise addition on T5, T6, T7, and T8 (i.e., ... Figure 9 The element-wise addition operation (+) is performed to obtain the multi-level fused feature map T9; then T9 is passed through the Sigmoid activation function to obtain the multi-level fused feature map Ti. 10 :

[0393] T 10 =S(DSC1(T1)+DSC3(T1)+DSC5(T1)+DSC7(T1))

[0394] In the above formula, DSC1() represents a depthwise separable convolution operation with a 1×1 kernel; DSC3() represents a depthwise separable convolution operation with a 3×3 kernel; DSC5() represents a depthwise separable convolution operation with a 5×5 kernel; DSC7() represents a depthwise separable convolution operation with a 7×7 kernel; + represents element-wise addition; S represents the Sigmoid activation function.

[0395] S32143: In the eighth branch, perform average pooling on T1 along the channel dimension (i.e., Figure 9 The channel-level average pooling operation (Channel Avg Pool) is performed on T1. Specifically, it iterates through each spatial position (i,j) of T1 (where i∈[1,H], j∈[1,W]) and calculates the average value of its position across all channels. In the ninth branch, a channel-level max pooling operation (i.e., max pooling) is performed on T1. Figure 9 The Channel Max Pool operation in T1 is as follows: Iterate through each spatial position (i,j) of T1 (where i∈[1,H], j∈[1,W]), calculate the maximum value of that position across all channels, and then concatenate the results from the eighth and ninth branches along the channel dimension (i.e., ...). Figure 9 The concatenation operation (Concat) is performed along the channel dimension to obtain a multi-level fused feature map T. 11 (Right now Figure 9 The convolution operation (Conv1×1) is performed in T; subsequently, a convolution kernel of size 1×1 is applied to T. 11 Perform convolution operations to obtain a multi-level fused feature map T. 12 Then apply a 7×7 convolution kernel to T 12 Perform convolution operations (i.e.) Figure 9 The convolution operation (Conv7×7) in the middle yields a multi-level fused feature map T. 13 Using a 3×3 convolution kernel with a dilation rate of 2 on T 12 Perform convolution operations (i.e.) Figure 9 The convolution operation (Conv3×3, Rate=2) yields a multi-level fused feature map T. 14 Then T 13 and T 14 Perform element-wise addition (i.e.) Figure 9 The element-wise addition operation (+) is used to obtain the multi-level fused feature map T. 15 ; then T 15 After applying the Sigmoid activation function, a multi-level fused feature map T is obtained. 16 :

[0396]

[0397] T 15 =S(Conv7(T) 12 )+Conv3_2(T 12 ))

[0398] In the above formula, T1(i,j,k) represents the eigenvalue of T1 at spatial position (i,j) and the k-th channel; max k∈[1,C]() indicates the operation of calculating the maximum value of each spatial location across all channels; Concat() indicates the concatenation operation along the channel dimension; Conv7() indicates the convolution operation with a 7×7 kernel; Conv3_2() indicates the convolution operation with a 3×3 kernel with a dilation of 2; S indicates the Sigmoid activation function; + indicates element-wise addition.

[0399] S32144: Place T1, T4, T 10 and T 16 Perform element-wise multiplication (i.e.) Figure 9 The element-wise multiplication operation (⊙) in the middle yields the multi-level fused feature map T. 17 Then apply a 1×1 convolution kernel to T. 17 Perform convolution operations to obtain a multi-level fused feature map T. 18 Then T 18 After applying the Softmax activation function, the anomaly mask M is obtained. a ;

[0400] After that, for M a Perform two sub-branch calculations; in the first sub-branch, calculate M. a The inverse anomaly mask (i.e.) Figure 9 The reverse anomaly mask operation RevMask is used to obtain the reverse anomaly mask M. 1-a Then T1 and M 1-a Perform element-wise multiplication (i.e.) Figure 9 The element-wise multiplication operation (⊙) in the middle yields the multi-level fused feature map T. 19 In the second sub-branch, T 17 and M a Perform element-wise multiplication (i.e.) Figure 9 The element-wise multiplication operation (⊙) in the middle yields the multi-level fused feature map T. 20 Finally, T 19 and T 20 Perform element-wise addition (i.e.) Figure 9 The element-wise addition operation (+) is used to obtain and output the multi-level fused feature map T. 21 ;

[0401] M a =Softmax(Conv1(T1⊙T4⊙T) 10 ⊙T 16 ))

[0402] M 1-a =1-M a

[0403] T 21 =T1⊙M 1-a+M a ⊙T 17

[0404] In the above formula, Conv1() represents a convolution operation with a kernel of size 1×1; Softmax represents the Softmax activation function; ⊙ represents element-wise multiplication; + represents element-wise addition.

[0405] Example:

[0406] Input a 14×14×256 Gram feature map T1 and distribute it into nine branches. In the first branch, no operation is performed on T1. In the second branch, no operation is performed on T1. In the third branch, T1 is first subjected to wavelet transform to extract high-frequency anomaly features, resulting in a multi-level fused feature map T2 with a size of 14×14×256. Then, T2 is convolved with a 1×1 kernel to obtain a multi-level fused feature map T3 with a size of 14×14×256. Finally, T3 is activated by a Sigmoid function to obtain a multi-level fused feature map T4 with a size of 14×14×256.

[0407] In the fourth branch, a depthwise separable convolution operation is performed on T1 using a 1×1 convolution kernel to obtain a multi-level fused feature map T5, with a size of 14×14×256. In the fifth branch, a depthwise separable convolution operation is performed on T1 using a 3×3 convolution kernel with a stride of 1 and padding of 1 to obtain a multi-level fused feature map T6, with a size of 14×14×256. In the sixth branch, a depthwise separable convolution operation is performed on T1 using a 5×5 convolution kernel with a stride of 1 and padding of 2 to obtain... The process begins with a multi-level fusion feature map T7, which has a size of 14×14×256. In the seventh branch, a 7×7 convolution kernel with a stride of 1 and padding of 3 is used to perform a depthwise separable convolution on T1, resulting in a multi-level fusion feature map T8, also with a size of 14×14×256. Then, T5, T6, T7, and T8 are added element-wise to obtain a multi-level fusion feature map T9, which also has a size of 14×14×256. Finally, T9 is activated by a sigmoid function to obtain the multi-level fusion feature map T0. 10 T 10 Its size is 14×14×256;

[0408] In the eighth branch, average pooling is performed on T1 along the channel dimension, with the result being 14×14×1. In the ninth branch, max pooling is performed on T1 along the channel dimension, with the result also being 14×14×1. The results from the eighth and ninth branches are then concatenated along the channel dimension to obtain the multi-level fused feature map T.11 T 11 The size is 14×14×2; subsequently, a convolution kernel of size 1×1 is applied to T. 11 Perform convolution operations to obtain a multi-level fused feature map T. 12 T 12 The size is 14×14×256; then a 7×7 convolution kernel is applied, with a stride of 1 and padding of 3, for T 12 Perform convolution operations to obtain a multi-level fused feature map T. 13 T 13 The size is 14×14×256; using a 3×3 convolution kernel with a dilation rate of 2, setting the stride to 1 and padding to 2, for T 12 Perform convolution operations to obtain a multi-level fused feature map T. 14 T 14 The size is 14×14×256; then T 13 and T 14 Perform element-wise addition to obtain the multi-level fused feature map T. 15 T 15 The size is 14×14×256; then T 15 After applying the Sigmoid activation function, a multi-level fused feature map T is obtained. 16 T 16 Its size is 14×14×256;

[0409] T1, T4, T 10 and T 16 Perform element-wise multiplication to obtain the multi-level fused feature map T. 17 T 17 The size is 14×14×256; then a 1×1 convolution kernel is applied to T. 17 Perform convolution operations to obtain a multi-level fused feature map T. 18 T 18 The size is 14×14×1; then T 18 After applying the Softmax activation function, the anomaly mask M is obtained. a M a The size is 14×14×1; then M a Perform two sub-branch calculations; in the first sub-branch, calculate M. a The inverse anomaly mask is obtained by using the inverse anomaly mask M. 1-a M 1-a The size is 14×14×1; then T1 and M 1-a Perform element-wise multiplication to obtain the multi-level fused feature map T. 19 T 19The size is 14×14×256; in the second sub-branch, T 17 and M a Perform element-wise multiplication to obtain the multi-level fused feature map T. 20 T 20 The size is 14×14×256; finally, T 19 and T 20 Perform element-wise addition to obtain and output the multi-level fused feature map T. 21 T 21 Its size is 14×14×256;

[0410] S3215: G 10 The input is fed into the Downsample module, which reduces its height and width by half while keeping the number of channels unchanged, resulting in the Gram extracted feature map G. 11 Then, G 11 The Gram feature map G is obtained by passing through 5 Bottleneck modules consecutively. 12 After that, G 12 The Gram feature map G is obtained through the MLAF module. 13 Next, G 13 Perform global average pooling (i.e.) Figure 7 The Global AVG Pooling operation in the middle yields the Gram extracted feature map G. 14 Finally, G 14 Perform layer normalization (i.e.) Figure 7 The LN operation in the process is used to obtain and output the Gram extracted feature map G. 15 ;

[0411] S32 Example:

[0412] The Gram angular field map G1 from the two-dimensional vibration subset obtained in S2 is used as the input to the Gram angular field map feature extraction network. G1 has a size of 224×224 pixels and 1 channel. G1 is then fed into a convolutional layer with a kernel size of 4×4 and a stride of 4, followed by an LN layer to obtain the Gram extracted feature map G2, which has a size of 56×56 pixels and 64 channels. G2 is then passed through three Bottleneck modules to obtain the Gram extracted feature map G3, which also has a size of 56×56 pixels and 64 channels. Finally, G3 is passed through a Downsample module to obtain the Gram extracted feature map G4, which has a size of 28×28 pixels and 128 channels.

[0413] Next, G4 is passed through 5 Bottleneck modules to obtain Gram extraction feature map G5, which is 28×28 pixels in size and has 128 channels. G5 is then input into the DAACO module to obtain Gram extraction feature map G6, which is 28×28 pixels in size and has 128 channels. G6 is then input into the Downsample module to obtain Gram extraction feature map G7, which is 14×14 pixels in size and has 256 channels.

[0414] Then, G7 is passed through 7 Bottleneck modules to obtain Gram extraction feature map G8, which is 14×14 pixels in size and has 256 channels. Next, G8 is passed through the DAACO module to obtain Gram extraction feature map G9, which is also 14×14 pixels in size and has 256 channels. Finally, G9 is input into the MLAF module to obtain Gram extraction feature map G... 10 G 10 The size is 14×14 pixels, and the number of channels is 256; G 10 The input is fed into the Downsample module to obtain the Gram extracted feature map G. 11 G 11 The size is 7×7 pixels, and the number of channels is 512;

[0415] Subsequently, G 11 The Gram feature map G is obtained by passing through 5 Bottleneck modules consecutively. 12 G 12 The size is 7×7 pixels, and the number of channels is 512; then, G 12 The Gram feature map G is obtained through the MLAF module. 13 G 13 The size is 7×7 pixels, and the number of channels is 512; next, G 13 Perform global average pooling to obtain the Gram extracted feature map G. 14 G 14 The size is 1×1 pixels, and the number of channels is 512; finally, G 14 Perform layer normalization to obtain and output the Gram extracted feature map G. 15 G 15 The size is 1×1 pixel, and the number of channels is 512;

[0416] S322: Extracting Temporal Features of Vibration Signals: The normalized vibration signals from the one-dimensional vibration subset are input into a bidirectional gated recurrent unit (BiGRU). First, the forward GRU layer in the BiGRU processes the vibration signals from left to right in chronological order, using a gating mechanism containing only update and reset gates to progressively update the forward hidden state and capture the forward temporal pattern of the vibration signals. Simultaneously, the reverse GRU layer processes the signals in reverse order from right to left, controlling the information flow through gating to capture the reverse dependencies of the vibration signals. Then, at each time step, the forward and reverse hidden states are concatenated along the channel dimension to form a fused feature containing bidirectional temporal information. Finally, global average pooling is performed on the fused features of all time steps to obtain the temporal feature map N1.

[0417] S323: G 15 The vibration feature map of the power equipment vibration signal is obtained by concatenating N1 with N1 along the feature dimension. Then, the vibration feature map of the power equipment vibration signal is classified by an MLP classifier to obtain a two-dimensional vector containing the logits scores of vibration with and without faults. Finally, the two-dimensional vector is normalized by the Softmax activation function to obtain the vibration prediction probability of faults. And vibration prediction fault-free probability

[0418] S33, Anomaly detection based on power equipment images: For power equipment images in image files, ADEI-Net is used to perform target detection on the power equipment images, extract the visual features of the equipment, and determine the anomalies in the images, outputting the probability of the image being predicted to have a fault and the probability of the image being predicted to be fault-free.

[0419] This invention designs an anomaly detection network for electrical equipment images called ADEI-Net (Anomaly Detection of Electrical Equipment Image Network), the overall structure of which is as follows: Figure 10 As shown;

[0420] The specific steps of S33 are as follows:

[0421] S331: Represent any power equipment image in the image subset as P1, input it into the CBS module to obtain the power equipment detection feature map P2; then, pass P2 through the CBS module to obtain the power equipment detection feature map P3; then, pass P3 through the C2F module to obtain the power equipment detection feature map P4; then, pass P4 through the CBS module to obtain the power equipment detection feature map P5.

[0422] The CBS module consists of three parts connected in series: a 3×3 Conv2d layer with a step size of 2, a BatchNorm2d layer, and a SiLU layer.

[0423] The C2F module consists of two 1×1 Conv2d modules with a step size of 1 and several Bottleneck modules. It introduces skip connections to connect feature maps at different stages, thereby promoting the transfer of features and the flow of information at different levels.

[0424] S332: Input P5 into the CFAE module to obtain the power equipment detection feature map P6;

[0425] This invention designs a Channel-Feature Adaptive Enhancement module (CFAE), the overall structure of which is as follows: Figure 11 As shown;

[0426] The power equipment detection feature map P5 is used as the input feature map for the CFAE module. For ease of description, P5 is referred to as V1 here;

[0427] The specific steps of S332 are as follows:

[0428] S3321: Input the power equipment detection feature map V1. First, use the channel shuffling method to perform a cross-mixing operation on the channel information of V1 (i.e., Figure 11 The channel shuffle operation (in the middle) is used to obtain the channel feature enhancement feature map V2; then, segmented GLU is used to perform gating processing on V2 (i.e., Figure 11 The GLU part of V2 is used to obtain the channel feature enhancement feature map V3. The specific steps are as follows: first, divide V2 into two parts along the channel dimension. 2_1 and V 2_2 (Right now Figure 11 The split operation in the middle), and then V 2_2 The gate signal G is obtained through the Sigmoid activation function, and finally V is... 2_1 Perform element-wise multiplication with G (i.e. Figure 11 The element-wise multiplication operation (⊙) is used to obtain V3; then, a convolution operation is performed on V3 using a convolution kernel of size 1×1 (i.e., ...). Figure 11 The convolution operation Conv1×1 in the middle is used to obtain the channel feature enhancement feature map V4;

[0429] S3322: Perform four-branch calculations on V4. In the first branch, global average pooling (i.e., ...) is first applied. Figure 11The Global Avg Pool operation in the middle is used to pool V4 to obtain the channel feature enhancement feature map V5; then a convolution operation of size 1×1 is applied to V5 (i.e., ...). Figure 11 The convolution operation Conv1×1 is performed on V4 to obtain the channel feature enhancement feature map V6; then V6 is passed through the Sigmoid activation function to obtain the channel feature enhancement feature map V7; in the second branch, a convolution operation of size 3×3 is performed on V4 (i.e., Figure 11 The convolution operation (Conv3×3) is performed to obtain the channel feature enhancement feature map V8; then V8 is passed through the Tanh activation function to obtain the channel feature enhancement feature map V9; in the third branch, no operation is performed on V4; then, V7, V9 and V4 are multiplied element-wise (i.e., ... Figure 11 The element-wise multiplication operation (⊙) yields the channel feature enhancement feature map V. 11 ;

[0430] S3323: In the fourth branch, V4 is processed by the Bottleneck module for feature extraction to obtain the channel feature enhancement feature map V. 10 Finally, V 11 and V 10 Perform channel-wise maximum value fusion (i.e.) Figure 11 The channel-wise maximum value fusion operation (MaxFusion) is used to obtain and output the channel feature enhancement feature map V. 12 The specific operation is as follows: For V 11 and V 10 For each corresponding channel, traverse all spatial locations on the feature map of that channel and calculate V. 11 and V 10 The maximum value of the element at that position, and use it as V. 12 Elements in;

[0431] Channel shuffling is a technique used to enhance the flow of information between feature channels. Its core idea is to rearrange the channels of a feature map, breaking the fixed grouping relationships between channels and thus promoting information exchange between different channels. Its working principle is as follows: Assuming the input feature map has C channels, it is divided into G groups, each containing C / G channels. First, the channel dimension of the input feature map is reshaped to (G, C / G, H, W). Then, the group dimension and channel dimension are swapped to obtain (C / G, G, H, W). Finally, it is flattened back to its original shape (C, H, W).

[0432] Example:

[0433] The input is a 28×28×256 power equipment detection feature map V1. First, the channel information of V1 is cross-mixed using the channel shuffling method to obtain a channel feature enhancement feature map V2, which is 28×28×256 in size. Then, V2 is gated using segmented GLU to obtain a channel feature enhancement feature map V3, which is 28×28×128 in size. The specific steps are: first, V2 is divided into two parts along the channel dimension V1 and V2. 2_1 and V 2_2 Both are 28×28×128 in size, and then V 2_2 The gate signal G is obtained through the Sigmoid activation function, and finally V is... 2_1 Element-wise multiplication is performed with G; then, a 1×1 convolution kernel is used to convolve V3 to obtain the channel feature enhancement feature map V4, which has a size of 28×28×256.

[0434] The computation proceeds through four branches for V4. In the first branch, global average pooling is applied to V4 to obtain channel feature enhancement feature map V5, which has a size of 1×1×256. Then, a 1×1 convolution kernel is applied to V5 to obtain channel feature enhancement feature map V6, which also has a size of 1×1×256. Next, V6 is processed through a sigmoid activation function to obtain channel feature enhancement feature map V7, which has a size of 1×1×256. In the second branch... In the second branch, a 3×3 convolution kernel with a stride of 1 and padding of 1 is used to convolve V4, resulting in a channel feature enhancement map V8 with a size of 28×28×256. V8 is then passed through a Tanh activation function to obtain a channel feature enhancement map V9 with a size of 28×28×256. In the third branch, no operation is performed on V4. Finally, V7, V9, and V4 are multiplied element-wise to obtain the channel feature enhancement map V. 11 V 11 Its size is 28×28×256;

[0435] In the fourth branch, V4 is processed by the Bottleneck module for feature extraction to obtain the channel feature enhancement feature map V. 10 V 10 The size is 28×28×256; finally, V 11 and V 10 Perform channel-wise maximum fusion to obtain and output channel feature enhancement feature map V. 12 V 12 Its size is 28×28×256;

[0436] S333: Input P6 into the CBS module to obtain the power equipment detection feature map P7; then, pass P7 through the CFAE module to obtain the power equipment detection feature map P8; next, pass P8 through the CBS module to obtain the power equipment detection feature map P9; finally, pass P9 through the CFAE module to obtain the power equipment detection feature map P... 10 Then P 10 The SPPF module is used to obtain the power equipment detection feature map P. 11 Subsequently, P 11 Perform upsampling operation (i.e.) Figure 10 The upsampling operation (Upsample) in the process yields the power equipment detection feature map P. 12 Next, P8 and P 12 splicing along the channel dimension (i.e.) Figure 10 The concatenation operation (Concat) is performed along the channel dimension to obtain the power equipment detection feature map P. 13 Then P 13 The power equipment detection feature map P is obtained through the C2F module. 14 ; then P 14 Perform upsampling operation (i.e.) Figure 10 The upsampling operation (Upsample) in the process yields the power equipment detection feature map P. 15 Next, P6 and P 15 splicing along the channel dimension (i.e.) Figure 10 The concatenation operation (Concat) is performed along the channel dimension to obtain the power equipment detection feature map P. 16 Then P 16 The power equipment detection feature map P is obtained through the C2F module. 17 ; then P 17 The power equipment detection feature map P is obtained through the CBS module. 18 Next, P 14 and P 18 splicing along the channel dimension (i.e.) Figure 10 The concatenation operation (Concat) is performed along the channel dimension to obtain the power equipment detection feature map P. 19 Then, P 19 The power equipment detection feature map P is obtained through the C2F module. 20 After that, P 20 The power equipment detection feature map P is obtained through the CBS module. 21 Then P 11 and P 21 splicing along the channel dimension (i.e.) Figure 10 The concatenation operation (Concat) is performed along the channel dimension to obtain the power equipment detection feature map P. 22Next, P 22 The power equipment detection feature map P is obtained through the C2F module. 23 ;

[0437] Among them, the SPPF module is an efficient multi-scale feature fusion module that quickly captures contextual information of different receptive fields through cascaded max pooling operations. This module first performs multiple max pooling operations of different sizes on the input feature map, then concatenates the original feature map with each pooling result in the channel dimension, and finally compresses the number of channels through 1×1 convolution. Compared with the traditional SPP module, SPPF adopts a serial stacked pooling method, which significantly reduces the amount of computation while maintaining the multi-scale feature extraction capability, and can effectively improve the model's detection performance for targets of different sizes.

[0438] S334: P 17 The data is input into the AADM module to obtain the power equipment detection feature map P. 24 ; P 20 The data is input into the AADM module to obtain the power equipment detection feature map P. 27 ; P 23 The data is input into the AADM module to obtain the power equipment detection feature map P. 30 ;

[0439] This invention designs an Attention Abnormality Detection module (AADM), the overall structure of which is as follows: Figure 12 As shown;

[0440] Power equipment detection feature map P 17 This serves as the input feature map for the AADM module. For ease of explanation, P will be referred to here. 17 It is denoted as Y1;

[0441] The specific steps of S334 are as follows:

[0442] S3341: Input the power equipment detection feature map Y1. First, perform a convolution operation on Y1 using a 3×3 convolution kernel (i.e., ... Figure 12 The convolution operation Conv3×3 in the middle uses a 5×5 convolution kernel to perform a convolution operation on Y1 (i.e. Figure 12 The convolution operation Conv5×5) and the convolution operation on Y1 using a convolution kernel of size 7×7 (i.e. Figure 12 The convolution operation (Conv7×7) is performed; then, the convolution results of the three operations are concatenated along the channel dimension (i.e., Figure 12 The concatenation operation (Concat) is performed along the channel dimension to obtain the attention anomaly detection feature map Y2; then, a convolution operation of size 1×1 is performed on Y2 (i.e., ...). Figure 12 The convolution operation Conv1×1 is performed to obtain the attention anomaly detection feature map Y3; then Y3 is first processed by the ReLU activation function, and then the result is processed by the Hard Sigmoid activation function to obtain the attention anomaly detection feature map Y4.

[0443] Y2=Concat(Conv3(Y1),Conv5(Y1),Conv7(Y1))

[0444] Y4 = HardS(ReLU(Conv1(Y2)))

[0445] In the above formula, Conv3() represents a convolution operation with a 3×3 kernel; Conv5() represents a convolution operation with a 5×5 kernel; Conv7() represents a convolution operation with a 7×7 kernel; Concat() represents a concatenation operation along the channel dimension; Conv1() represents a convolution operation with a 1×1 kernel; ReLU represents the ReLU activation function; HardS represents the Hard Sigmoid activation function.

[0446] S3342: Perform element-wise multiplication of Y4 and Y1 (i.e. Figure 12 The element-wise multiplication operation (⊙) is used to obtain the attention anomaly detection feature map Y5; then, a depthwise separable convolution operation (i.e., ⊙) is performed on Y5 using a 3×3 convolution kernel. Figure 12 The depthwise separable convolution operation (DSC3×3) is used to obtain the attention anomaly detection feature map Y6; then Y6 is passed through the Sigmoid activation function to obtain the attention anomaly detection feature map Y7; finally, Y7 and Y5 are multiplied element-wise (i.e., ... Figure 12 The element-wise multiplication operation ⊙) is used to obtain the attention anomaly detection feature map Y8;

[0447] Y8=(Y4⊙Y1)⊙S(DSC3(Y5))

[0448] In the above formula, DSC3() represents a depthwise separable convolution operation with a 3×3 kernel; S represents the Sigmoid activation function; ⊙ represents element-wise multiplication.

[0449] S3343: Use a fully connected layer to perform feature mapping on Y8 (i.e., ... Figure 12 The FC module in the code obtains the attention anomaly detection feature map Y9; then, Y9 is processed by the Sigmoid activation function to obtain the attention anomaly detection feature map Yi. 10 Finally, Y 10 Perform element-wise multiplication with Y8 (i.e.) Figure 12The element-wise multiplication operation ⊙) is used to obtain the attention anomaly detection feature map Y. 11 ;

[0450] Y 11 =S(FC(Y8))⊙Y8

[0451] In the above formula, S represents the Sigmoid activation function; FC() represents the feature mapping operation of the fully connected layer; ⊙ represents the element-wise multiplication operation;

[0452] Example:

[0453] First, input a power equipment detection feature map Y1 with a size of 28×28×256. Then, perform convolution operations on Y1 using a 3×3 kernel (stride 1, padding 1), a 5×5 kernel (stride 1, padding 2), and a 7×7 kernel (stride 1, padding 3). Next, concatenate the three convolution results along the channel dimension to obtain the attention anomaly detection feature map Y2, with a size of 28×28×768. Then, perform a convolution operation on Y2 using a 1×1 kernel to obtain the attention anomaly detection feature map Y3, with a size of 28×28×256. Finally, process Y3 first with a ReLU activation function and then with a Hard Sigmoid activation function to obtain the attention anomaly detection feature map Y4, with a size of 28×28×256.

[0454] Next, Y4 and Y1 are multiplied element-wise to obtain the attention anomaly detection feature map Y5, which has a size of 28×28×256. Then, a depthwise separable convolution operation is performed on Y5 using a 3×3 convolution kernel with a stride of 1 and padding of 1, resulting in the attention anomaly detection feature map Y6, which also has a size of 28×28×256. Y6 is then activated by the Sigmoid function to obtain the attention anomaly detection feature map Y7, which also has a size of 28×28×256. Finally, Y7 and Y5 are multiplied element-wise to obtain the attention anomaly detection feature map Y8, which also has a size of 28×28×256.

[0455] Subsequently, a fully connected layer is used to perform feature mapping on Y8 to obtain the attention anomaly detection feature map Y9, which has a size of 28×28×256. Then, Y9 is processed through a sigmoid activation function to obtain the attention anomaly detection feature map Y0. 10 Y 10 The size is 28×28×256; finally, Y 10 Element-wise multiplication with Y8 yields the attention anomaly detection feature map Y. 11 Y 11 Its size is 28×28×256;

[0456] S335: P 24 The data is input into the AADM module to obtain the power equipment detection feature map P. 25 ; P 27 The data is input into the AADM module to obtain the power equipment detection feature map P. 28 ; P 30 The data is input into the AADM module to obtain the power equipment detection feature map P. 31 ;

[0457] After that, P 25 Through a convolutional layer with a kernel size of 1×1 (i.e. Figure 10 The Conv2d module in the middle obtains the power equipment detection feature map P. 26 ; P 28 Through a convolutional layer with a kernel size of 1×1 (i.e. Figure 10 The Conv2d module in the middle obtains the power equipment detection feature map P. 29 ; P 31 Through a convolutional layer with a kernel size of 1×1 (i.e. Figure 10 The Conv2d module in the middle obtains the power equipment detection feature map P. 32 ;

[0458] Subsequently, for P 26 The values ​​at all locations in the spatial dimension (i.e., the height and width directions) are averaged (i.e., Figure 10 The SpatialAvg operation in the image yields a two-dimensional vector X1 containing the faulty logits score and the fault-free logits score of the image; for P 29 The values ​​at all locations in the spatial dimension (i.e., the height and width directions) are averaged (i.e., Figure 10 The SpatialAvg operation in the image yields a two-dimensional vector X2 containing the faulty and fault-free logits scores of the image; P 32 The values ​​at all locations in the spatial dimension (i.e., the height and width directions) are averaged (i.e., Figure 10 The SpatialAvg operation in the image is used to obtain a two-dimensional vector X3 containing the faulty and fault-free logits scores of the image; then, the three levels of two-dimensional vectors are added element-wise (i.e., ...). Figure 10 The element-wise addition operation (+) is performed to obtain the comprehensive logits vector X4; finally, the comprehensive logits vector is transformed into a probability distribution through the Softmax activation function, resulting in a two-dimensional vector P containing the probability of image prediction failure and the probability of image prediction failure-free. out Then according to P out The probability of failure is obtained from image prediction. And image prediction fault-free probability

[0459] X4 = X1 + X2 + X3

[0460] P out =Softmax(X4)

[0461]

[0462] In the above formula, Softmax represents the Softmax activation function; + indicates element-wise addition.

[0463] Where: logits is the unnormalized score used to calculate the class probability;

[0464] S33 Example:

[0465] The power equipment image P1 from the image subset obtained in S2 is used as the input to the power equipment image anomaly detection network. P1 has a size of 224×224 pixels and 3 channels. P1 is then input to the CBS module to obtain the power equipment detection feature map P2, which has a size of 112×112 pixels and 64 channels. Next, P2 is passed through the CBS module to obtain the power equipment detection feature map P3, which has a size of 56×56 pixels and 128 channels. Then, P3 is passed through the C2F module to obtain the power equipment detection feature map P4, which also has a size of 56×56 pixels and 128 channels. Finally, P4 is passed through the CBS module to obtain the power equipment detection feature map P5. The first image (P5) is 28×28 pixels in size and has 256 channels. Inputting P5 into the CFAE module yields the power equipment detection feature map P6, also 28×28 pixels in size and with 256 channels. Inputting P6 into the CBS module yields the power equipment detection feature map P7, 14×14 pixels in size and with 512 channels. Then, passing P7 through the CFAE module yields the power equipment detection feature map P8, also 14×14 pixels in size and with 512 channels. Passing P8 through the CBS module yields the power equipment detection feature map P9, 7×7 pixels in size and with 1024 channels. Finally, passing P9 through the CFAE module yields the power equipment detection feature map P... 10 P 10 The size is 7×7 pixels, and the number of channels is 1024; then P 10 The SPPF module is used to obtain the power equipment detection feature map P. 11 P 11 The size is 7×7 pixels, and the number of channels is 1024;

[0466] Then, P 11An upsampling operation is performed to obtain the power equipment detection feature map P. 12 P 12 The size is 14×14 pixels, and the number of channels is 512; next, P8 and P... 12 By stitching along the channel dimension, the power equipment detection feature map P is obtained. 13 P 13 The size is 14×14 pixels, and the number of channels is 1024; then P 13 The power equipment detection feature map P is obtained through the C2F module. 14 P 14 The size is 14×14 pixels, and the number of channels is 512; then P 14 Through upsampling, the power equipment detection feature map P is obtained. 15 P 15 The size is 28×28 pixels, and the number of channels is 256; next, P6 and P 15 By stitching along the channel dimension, the power equipment detection feature map P is obtained. 16 P 16 The size is 28×28 pixels, and the number of channels is 512; then P 16 The power equipment detection feature map P is obtained through the C2F module. 17 P 17 The size is 28×28 pixels, and the number of channels is 256; then P 17 The power equipment detection feature map P is obtained through the CBS module. 18 P 18 The size is 28×28 pixels, and the number of channels is 256; next, P... 14 and P 18 By stitching along the channel dimension, we obtain the power equipment detection feature map P. 19 P 19 The size is 28×28 pixels, and the number of channels is 768; then, P 19 The power equipment detection feature map P is obtained through the C2F module. 20 P 20 The size is 28×28 pixels, and the number of channels is 256; then, P 20 The power equipment detection feature map P is obtained through the CBS module. 21 P 21 The size is 28×28 pixels, and the number of channels is 256; then P 11 and P 21 By stitching along the channel dimension, the power equipment detection feature map P is obtained. 22 P 22 The size is 28×28 pixels, and the number of channels is 1280; next, P 22The power equipment detection feature map P is obtained through the C2F module. 23 P 23 The size is 28×28 pixels, and the number of channels is 256;

[0467] Next, P 17 The data is input into the AADM module to obtain the power equipment detection feature map P. 24 P 24 The size is 28×28 pixels, and the number of channels is 256; P 20 The data is input into the AADM module to obtain the power equipment detection feature map P. 27 P 27 The size is 28×28 pixels, and the number of channels is 256; P 23 The data is input into the AADM module to obtain the power equipment detection feature map P. 30 P 30 The size is 28×28 pixels, and the number of channels is 256; then, P 24 The data is input into the AADM module to obtain the power equipment detection feature map P. 25 P 25 The size is 28×28 pixels, and the number of channels is 256; P 27 The data is input into the AADM module to obtain the power equipment detection feature map P. 28 P 28 The size is 28×28 pixels, and the number of channels is 256; P 30 The data is input into the AADM module to obtain the power equipment detection feature map P. 31 P 31 The size is 28×28 pixels, and the number of channels is 256; then, P 25 The power equipment detection feature map P is obtained by using a convolutional layer with a kernel size of 1×1. 26 P 26 The size is 28×28 pixels, and the number of channels is 2; P 28 The power equipment detection feature map P is obtained by using a convolutional layer with a kernel size of 1×1. 29 P 29 The size is 28×28 pixels, and the number of channels is 2; P 31 The power equipment detection feature map P is obtained by using a convolutional layer with a kernel size of 1×1. 32 P 32 The size is 28×28 pixels, and the number of channels is 2;

[0468] Subsequently, for P 26 The values ​​at all locations along the spatial dimensions (i.e., height and width) are averaged to obtain a two-dimensional vector X1 containing the faulty logits score and the fault-free logits score of the image; for P29 The values ​​at all locations along the spatial dimensions (i.e., height and width) are averaged to obtain a two-dimensional vector X2 containing the faulty and fault-free logits scores of the image; P 32 The values ​​at all locations in the spatial dimensions (i.e., height and width) are averaged to obtain a two-dimensional vector X3 containing the faulty and fault-free logits scores of the image. Then, these three two-dimensional vectors are summed element-wise to obtain a comprehensive logits vector X4. Finally, the comprehensive logits vector is transformed into a probability distribution using a softmax activation function, outputting the probability of image fault prediction. And image prediction fault-free probability

[0469] S34, Determine if the power equipment is faulty: Combine the predicted probabilities of each mode, and then perform decision-level fusion through the multimodal weighted fusion (MMWF) algorithm to obtain the overall predicted probability of fault and the overall predicted probability of no fault; and determine whether the power equipment is faulty based on the probability values; if the overall predicted probability of fault is greater than the overall predicted probability of no fault, then the output equipment is faulty; otherwise, the output equipment is faultless.

[0470] This invention designs a multi-modal weighted fusion algorithm, MMWF (Multi-Modal Weighted Fusion Algorithm), the overall structure of which is as follows: Figure 13 As shown;

[0471] The specific steps of S34 are as follows:

[0472] The input sub-model M1 has a fault probability for audio prediction of power equipment. And audio prediction fault-free probability Sub-model M2 has a failure probability for vibration prediction of power equipment. And vibration prediction fault-free probability Sub-model M3 has a fault probability for image prediction of power equipment. And image prediction fault-free probability First, calculate the mutual information I(M1,M2) between M1 and M2, the mutual information I(M1,M3) between M1 and M3, and the mutual information I(M2,M3) between M2 and M3. Then, based on the mutual information calculation results, adjust the weights of each sub-model using a weighted average to obtain the adjusted weight w of model M1. adjust (1) M2 model weights w adjust (2) and the weights w of the M3 model adjust(3); Then apply the weighted voting method to calculate the probability P of a faulty weighted vote. vote (1) and the fault-free weighted voting probability P vote (2); Then, the soft voting method is applied to calculate the probability P of a faulty soft vote. soft (1) and the probability of fault-free soft voting P soft (2); Finally, the results of the weighted fusion of the weighted voting method and the soft voting method are used to obtain the final prediction result: the overall prediction probability of failure P. mmwf (1) and the overall predicted fault-free probability P mmwf (2);

[0473]

[0474]

[0475] P mmwf (y)=α*P vote (y)+(1-α)*P soft (y)

[0476] In the above formula, I(M) i M j ) indicates the calculation of M i and M j Mutual information between them; y represents a binary classification value indicating whether there is a fault, where y=1 indicates a fault and y=2 indicates no fault; Submodel M i The predicted probability of category y; P(y) is the prior probability of category y, where P(y=1) equals the ratio of the number of faulty samples in the dataset to the total number of samples, and P(y=2) equals the ratio of the number of fault-free samples in the dataset to the total number of samples; w i Each sub-model M represents i Learnable weights; k represents the number of sub-models; w adjust (i) represents the sub-model M adjusted based on mutual information. i Normalized weights; 1(y=y pred,i ) represents the sub-model M i The indicator function for whether the prediction is for category y; α represents a learnable parameter used to balance the effects of weighted voting and soft voting;

[0477] The weighted voting method is a multi-model decision fusion strategy that assigns different weights to each sub-model to reflect the differences in the reliability of their prediction results. The specific process is as follows: First, weight coefficients are set for each sub-model (such as audio, vibration, and image analysis models) based on validation set performance or prior knowledge. Then, the failure probabilities output by each model are summed according to their weights, and finally, the equipment status is determined based on the weighted score. For example, if the image model has a higher classification accuracy, it can be given a larger weight (e.g., 0.5), while the vibration and audio models have weights of 0.3 and 0.2, respectively. This method highlights the contribution of high-confidence models through weight differentiation and is suitable for scenarios where the performance of each sub-model is significantly unbalanced.

[0478] Soft voting is an algorithm that achieves multi-model fusion by directly averaging the probability distributions output by each sub-model. Its core idea is that the predicted probabilities of all sub-models are equally important. The arithmetic mean of the failure probability and normal probability output by each model is calculated, and the average probability is used as the basis for comprehensive judgment. For example, if the failure probabilities predicted by three sub-models are 0.7, 0.8, and 0.6 respectively, the fusion result is (0.7 + 0.8 + 0.6) / 3 = 0.7. This method does not require weight setting, emphasizes "collective decision-making," and is suitable for situations where the performance of each sub-model is similar and the prediction results are complementary. It can effectively reduce the impact of random errors from individual models.

[0479] Example:

[0480] Input sub-models M1, M2, and M3 respectively predict the probability of faults in power equipment. and the probability of fault-free prediction Prior probabilities: P(y=1) = 0.5, P(y=2) = 0.5 (assuming a balance between positive and negative samples in the dataset); w i Take 1 / 3, and α = 0.5;

[0481] The first step is to calculate the mutual information between the models.

[0482]

[0483] Similarly, we can calculate that: I(M1,M3)≈0.3829, I(M2,M3)≈0.2819

[0484] The second step is to calculate the adjusted model weights.

[0485]

[0486] Similarly, we can calculate:

[0487] w adjust (2)≈0.3263

[0488] wadjust (3)≈0.3276

[0489] The third step is to apply weighted voting.

[0490] P vote (1) = w adjust (1)*1(y=1)+w adjust (3)*1(y=1)

[0491] =0.3461 + 0.3276 = 0.6737

[0492] P vote (2) = w adjust (2)*1(y=2)=0.3263

[0493] The fourth step is to apply the soft voting method.

[0494]

[0495] Step 5: Calculate the MMWF prediction results

[0496] Probability of failure:

[0497] P mmwf (1)=α*P vote (y=1)+(1-α)*P soft (y=1)

[0498] =0.5*0.6737+0.5*0.6004=0.6371

[0499] Probability of no failure:

[0500] P mmwf (2)=α*P vote (y=2)+(1-α)*P soft (y=2)=0.5*0.3263+0.5*0.3996=0.3630;

[0501] S4: Divide the preprocessed multimodal dataset of power equipment, and then train and validate the power equipment anomaly detection model;

[0502] The specific steps of S4 are as follows:

[0503] S41: Divide the preprocessed multimodal dataset of power equipment in S2 into training set, validation set and test set according to a certain ratio to ensure that the samples in the training set, validation set and test set are independent and do not overlap; during the partitioning process, ensure that the class distribution of each subset is consistent with the overall dataset to avoid the negative impact of data imbalance on model training;

[0504] S42: A dynamically adjusted hyperparameter scheme is adopted: the basic training cycle is set to a certain number of rounds, and a progressive batch expansion strategy is implemented, that is, the initial batch size is set to a small value, and the batch size is gradually increased after a certain number of training cycles; the learning rate scheduling adopts the cyclic cosine annealing algorithm, starting from a low baseline value, and setting the cycle length according to the training requirements, so that the learning rate decreases smoothly during the training process, effectively improving the model convergence; the optimizer is NAdam, and gradient normalization constraints are set to prevent the gradient from being too large by limiting the L2 norm of parameter updates, thus ensuring the stability of the training process;

[0505] S43: During the training phase, the input samples in each batch are a quintuple, which includes: normalized audio data from the one-dimensional sound subset, Mel spectrograms from the two-dimensional sound subset, normalized vibration signals from the one-dimensional vibration subset, Gram angular field maps from the two-dimensional vibration subset, and images of power equipment from the image subset. Through the quintuple samples, the model can learn the correlation between different modalities. During feature extraction, a cross-modal contrastive loss function is used to optimize the model, enabling data from different modalities to be effectively aligned in the feature space, thereby improving multimodal learning capabilities.

[0506] S44: During the validation phase, a multi-dimensional evaluation protocol is adopted, and a full validation is performed every few batches. The validation process includes several test modes: standard data inference, noise interference test, and modality missing test. An adaptive early stopping mechanism is set up: when the coefficient of variation of the main evaluation metric remains below the preset value in several consecutive validations, the training is automatically terminated to ensure the convergence and stability of the model.

[0507] S45: During the testing phase, the model will undergo several rigorous tests on the test set, including: cross-device generalization evaluation, real-time stress testing, model stability, robustness, and performance evaluation under data noise interference; through testing, it is ensured that the model can maintain high reliability and accuracy in practical applications and meet the actual needs of power equipment anomaly detection.

[0508] Example:

[0509] S41 divides the preprocessed multimodal dataset of power equipment in S2 into training set, validation set and test set in a ratio of 7:2:1, resulting in a training set containing 7000 data points, a validation set containing 2000 data points and a test set containing 1000 data points; at the same time, the division process ensures that the category distribution of each subset is consistent with the overall dataset.

[0510] S42 employs a dynamically adjusted hyperparameter scheme: the basic training cycle is set to 120 epochs; a progressive batch expansion strategy is implemented, with an initial batch size of 32, which doubles every 30 training epochs; the learning rate scheduling uses the cyclic cosine annealing algorithm, with a baseline value of 0.002 and a cycle length of 40 training epochs; the optimizer is NAdam, and gradient normalization constraints are set, with the upper limit of the L2 norm for parameter updates set to 2.0;

[0511] During the training phase, the input samples in each batch of S43 are a quintuple, which includes: normalized audio data from the one-dimensional sound subset, Mel spectrograms from the two-dimensional sound subset, normalized vibration signals from the one-dimensional vibration subset, Gram angular field maps from the two-dimensional vibration subset, and images of power equipment from the image subset. During feature extraction, a cross-modal contrastive loss function is used to optimize the model, so that data from different modalities can be effectively aligned in the feature space.

[0512] During the validation phase, S44 employs a multi-dimensional evaluation protocol, performing a full validation every 5 batches. The validation process includes three test modes: standard data inference (for basic performance evaluation), noise interference test (Gaussian noise is added to the data to simulate environmental noise), and modality missing test (randomly masking some modal inputs). An adaptive early stopping mechanism is also set up: training is automatically terminated when the coefficient of variation of the main evaluation metric remains below 5% for 15 consecutive validations, ensuring model convergence stability.

[0513] During the testing phase, the S45 model underwent five rigorous tests on 1,000 test sets, including but not limited to: cross-device generalization assessment, real-time stress testing, model stability, robustness, and performance evaluation under data noise interference, to ensure that the model can maintain high reliability and accuracy in practical applications and meet the actual needs of power equipment anomaly detection.

[0514] S5: Apply the trained power equipment anomaly detection model to detect power equipment;

[0515] The specific steps of S5 are as follows:

[0516] S51: Input the collected audio data, vibration signals and images of the power equipment into the power equipment anomaly detection model to obtain the overall predicted probability of failure and the overall predicted probability of no failure.

[0517] S52: Then make a judgment. If the overall predicted probability of failure is greater than the overall predicted probability of no failure, then the output device is faulty; otherwise, the output device is not faulty.

[0518] In summary, this invention extracts multimodal anomaly features from audio data, vibration signals, and images of power equipment and performs multimodal fusion to detect faults in power equipment; specifically, it includes:

[0519] (1) A Mel spectrogram feature extraction network MSFE-Net was designed to extract the frequency domain features of audio data; BiLSTM was used to extract the temporal domain features of audio data, and then the frequency domain features and temporal domain features of audio data were fused to fully capture the complex information of audio data; a spatial adaptive pooling module and a local-global fusion attention module were designed in MSFE-Net to retain and enhance the abnormal features in audio data.

[0520] (2) A Gram field map feature extraction network GAFE-Net was designed to extract the spatial features of vibration signals; BiGRU was used to extract the temporal features of vibration signals, and then the spatial and temporal features of vibration signals were fused to extract the abnormal information of vibration signals under complex environments; a dynamic abnormal perception channel optimization module and a multi-level adaptive fusion module were designed in GAFE-Net to improve the model's ability to identify abnormal vibration signals under complex environments.

[0521] (3) An ADEI-Net network for detecting anomalies in power equipment images was designed to process power equipment images and output the probability of failure and the probability of no failure in the image of the power equipment. A channel feature adaptive enhancement module and an attention anomaly detection module were designed in ADEI-Net to enhance the feature extraction capability of abnormal regions in the image and improve the detection accuracy of image anomalies.

[0522] (4) A multimodal weighted fusion algorithm was designed. This algorithm introduces mutual information into the weighted voting method and the soft voting method. It can automatically adjust the weight according to the contribution of each mode in the power equipment anomaly detection task, so as to ensure that the fusion result can make full use of the information of each mode, thereby improving the overall effect of power equipment anomaly detection.

[0523] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A power equipment anomaly detection method based on multimodal deep learning, characterized in that, The specific steps are as follows: S1: Construct a raw multimodal dataset of power equipment that includes audio data, vibration signals, and images of power equipment; S2: Preprocess the original power equipment multimodal dataset to obtain the preprocessed power equipment multimodal dataset; S3: Construct an anomaly detection model for power equipment based on multimodal deep learning; S4: Divide the preprocessed multimodal dataset of power equipment, and then train and validate the power equipment anomaly detection model; S5: Apply the trained power equipment anomaly detection model to detect power equipment; The specific steps of S3 are as follows: S31, Anomaly detection based on audio data of power equipment: First, the frequency domain features of the audio data are extracted from the Mel spectrogram of the two-dimensional sound subset of S2 using MSFE-Net; and the time domain features of the audio data are extracted from the normalized audio data of the one-dimensional sound subset of S2 using BiLSTM. Subsequently, the results from both are fused using time-frequency co-feature fusion at the feature layer to obtain the audio feature map of the power equipment audio data; then, the audio feature map of the power equipment audio data is classified using an MLP classifier; finally, the classification results are converted into audio prediction of fault probability and audio prediction of no fault probability using a Softmax activation function. S32, Anomaly Detection Based on Vibration Signals of Power Equipment: First, the Gram angular field map in the two-dimensional vibration subset of S2 is used to extract the spatial features of the vibration signal through GAFE-Net; and the normalized vibration signal in the one-dimensional vibration subset of S2 is used to extract the temporal features of the vibration signal through BiGRU. Subsequently, the results from both are fused temporally at the feature layer to obtain the vibration feature map of the power equipment vibration signal; then, the vibration feature map of the power equipment vibration signal is classified by an MLP classifier; finally, the classification results are converted into vibration prediction fault probability and vibration prediction no-fault probability through a Softmax activation function. S33, Anomaly detection based on power equipment images: For power equipment images in image files, ADEI-Net is used to perform target detection on the power equipment images, extract the visual features of the equipment, and determine the anomalies in the images, outputting the probability of the image being predicted to have a fault and the probability of the image being predicted to be fault-free. S34, Determine if the power equipment is faulty: Combine the predicted probabilities of each mode, and then perform decision-level fusion through a multimodal weighted fusion algorithm to obtain the overall predicted probability of fault and the overall predicted probability of no fault; and determine whether the power equipment is faulty based on the probability values; if the overall predicted probability of fault is greater than the overall predicted probability of no fault, then the output equipment is faulty; otherwise, the output equipment is faultless.

2. The power equipment anomaly detection method based on multimodal deep learning according to claim 1, characterized in that, The specific steps of S1 are as follows: S11: First, use a sound sensor to collect audio data generated by the power equipment during operation and save it as an audio file; then, use a vibration sensor to collect vibration signals of the power equipment and save them as vibration data files; next, use a high-definition camera to capture images of the working area of ​​the power equipment, record images of the power equipment covering key components, and save them as image files. S12: Label the acquired audio files, vibration data files, and image files; determine whether the current device has a fault from the audio data in the audio files, and then label the audio data as either faulty or fault-free; determine whether the current device has a fault from the vibration signals in the vibration data files, and then label the vibration signals as either faulty or fault-free; determine whether the current device has a fault from the power equipment images in the image files, and then use the image labeling tool to label the power equipment images as either faulty or fault-free.

3. The power equipment anomaly detection method based on multimodal deep learning according to claim 1, characterized in that, The specific steps of S2 are as follows: S21: Normalize the audio data in the audio file, that is, unify the sampling frequency and sampling duration, so that the audio data in the audio file has the same sampling interval, and obtain a one-dimensional sound subset. S22: Process the audio data in the audio file into a Mel spectrogram using Mel frequency cepstral coefficients to obtain a two-dimensional sound subset; S23: Normalize the vibration signals in the vibration data file to make their amplitudes within a uniform range, and then use the sliding window method to divide the vibration data into multiple time periods, each containing vibration data of uniform length, to obtain a one-dimensional vibration subset. S24: Convert the vibration signals in the vibration data file into Gram angular field diagrams to obtain a two-dimensional vibration subset data set; S25: Adjust the power equipment images in the image file to a uniform size, then use data augmentation techniques to generate multiple augmented images and add them to the dataset to obtain an image subset; S26: Perform time alignment operation: For each sampling time point, extract the data samples corresponding to the time from all subsets, including the normalized audio data in the one-dimensional sound subset, the Mel spectrogram in the two-dimensional sound subset, the normalized vibration signal in the one-dimensional vibration subset, the Gram angle field map in the two-dimensional vibration subset, and the power equipment image in the image subset, and combine them to form a quintuple sample. The process achieves cross-modal synchronization through timestamp matching, ensuring that the data samples of all subsets are aligned in the time dimension; this alignment operation is repeated for all available time points to obtain the preprocessed power equipment multimodal dataset.

4. The power equipment anomaly detection method based on multimodal deep learning according to claim 1, characterized in that, The specific steps of S31 are as follows: S311: Extract the frequency domain features of the audio data from the Mel spectrograms in the two-dimensional sound subset using the Mel spectrogram feature extraction network MSFE-Net, and obtain the Mel high-level feature map F. 19 ; S312: Extracting temporal features of audio data: Inputting the normalized audio data from the one-dimensional sound subset into a bidirectional long short-term memory network (BiLSTM); First, the forward LSTM layer in BiLSTM processes the audio data from left to right step by step, gradually updating the forward hidden state to capture the forward dependencies of the audio data in the time series; at the same time, the backward LSTM layer in BiLSTM processes the audio data from right to left step by step, updating the backward hidden state to capture the backward dependencies of the audio data. Next, at each time step, the forward and backward hidden states are concatenated to obtain the bidirectional features of the time step; finally, global average pooling is performed on the bidirectional features of all time steps to obtain the temporal feature map E1. S313: F 19 The audio feature map of the power equipment audio data is obtained by concatenating E1 along the feature dimension. Then, an MLP classifier is used to classify the audio feature map of the power equipment audio data, resulting in a two-dimensional vector containing the audio fault logits score and the audio fault-free logits score. Finally, the two-dimensional vector is normalized using the Softmax activation function to obtain the audio fault prediction probability. And audio prediction fault-free probability The specific steps of S311 are as follows: S3111: Represent any Mel spectrogram in the two-dimensional sound subset as F1, input it into a convolutional layer with 64 3×3 kernels and a stride of 1, and then pass it through the ReLU activation function and batch normalization to obtain the Mel shallow feature map F2; then, input F2 into a convolutional layer with 64 3×3 kernels and a stride of 1, and then pass it through the ReLU activation function and batch normalization to obtain the Mel shallow feature map F3; S3112: Input F3 into the SAPM module to obtain the Mel shallow feature map F4; where F3 is denoted as A1; S3113: Input F4 into a convolutional layer with 128 3×3 kernels and a stride of 1, then pass it through the ReLU activation function and batch normalization to obtain the Mel shallow feature map F5; then input F5 into a convolutional layer with 128 3×3 kernels and a stride of 1, then pass it through the ReLU activation function and batch normalization to obtain the Mel shallow feature map F6; finally, input F6 into the SAPM module to obtain the Mel shallow feature map F7; S3114: Input F7 into a convolutional layer with 256 3×3 kernels and a stride of 1, then pass it through ReLU activation and batch normalization to obtain the Mel-Ray mid-layer feature map F8; then input F8 into a convolutional layer with 256 3×3 kernels and a stride of 1, then pass it through ReLU activation and batch normalization to obtain the Mel-Ray mid-layer feature map F9; finally, input F9 into the SAPM module to obtain the Mel-Ray mid-layer feature map F... 10 ; S3115: F 10 The input is fed into a convolutional layer with 512 3×3 kernels and a stride of 1, and then sequentially processed by ReLU activation and batch normalization to obtain the Mel high-level feature map F. 11 Then F 11 The input is fed into a convolutional layer with 512 3×3 kernels and a stride of 1, and then sequentially processed by ReLU activation and batch normalization to obtain the Mel high-level feature map F. 12 ; S3116: F 12 The input is fed into the LGFA module to obtain the Mel high-level feature map F. 13 ; S3117: F 13 The input is fed into the SAPM module to obtain the Mel high-level feature map F. 14 Next, F 14 The input is fed into a convolutional layer with 512 3×3 kernels and a stride of 1, and then sequentially processed by ReLU activation and batch normalization to obtain the Mel high-level feature map F. 15 Then, F 15 The input is fed into a convolutional layer with 512 3×3 kernels and a stride of 1, and then sequentially processed by ReLU activation and batch normalization to obtain the Mel high-level feature map F. 16 After that, F 16 The input is fed into the LGFA module to obtain the Mel high-level feature map F. 17 Next, F 17 The input is fed into the SAPM module to obtain the Mel high-level feature map F. 18 Finally, F 18 The input is fed into a fully connected layer, and the high-level feature map F of Mel is obtained and output. 19 ; The specific steps of S3112 are as follows: S31121: In the first branch, calculate A along the vertical axis. 1_p The mean value is used to obtain the vertical mean vector μ. p,h In the second branch, calculate A along the vertical axis. 1_p The variance is used to obtain the vertical variance vector. m p,h =[μ p,h [1],m p,h [2],…,m p,h [h]] In the above formula, A 1_p (i,j) represents the state in A. 1_p The eigenvalue at position (i,j) in the middle; S31122: In the third branch, according to A 1_p Calculate the height H to output the height H 2_p ; S31123: In the fourth branch, A is not correct. 1_p Perform any operation; in the fifth branch, calculate A along the horizontal axis. 1_p The mean value is used to obtain the water average vector μ. p,w In the sixth branch, calculate A along the horizontal axis. 1_p The variance is used to obtain the level variance vector. m p,w =[μ p,w [1],m p,w [2],…,m p,w [w]] In the above formula, A 1_p (i,j) represents the state in A. 1_p The eigenvalue at position (i,j) in the middle; S31124: In the seventh branch, according to A 1_p The width W is used to calculate the output width W. 2_p ; S31125: Based on variance and Calculations are performed to obtain the variance factor. Then according to μ p,h Variance-guided fusion calculation was performed to obtain the mean μ. h Then according to Will Variance-guided fusion calculation is performed to obtain the variance. Then according to μ p,w Variance-guided fusion calculation was performed to obtain the mean μ. w Then according to Will Variance-guided fusion calculation was performed to obtain the mean. Subsequently, according to μ h and The calculation is performed to obtain the vertical pooling window size k. h According to μ w and The calculation is performed to obtain the horizontal pooling window size k. w ; In the above formula, max() represents the operation of calculating the maximum value; mean() represents the operation of calculating the average value; round() represents the operation of rounding the number to the nearest integer; k min ε is the size of the minimum pooling window, used to avoid the pooling window being too small; ε is a constant used to prevent instability caused by the mean being close to zero or the variance being too small. S31126: According to k h and H 2_p Calculate the vertical step size s h Then, according to k w and W 2_p Calculate the horizontal step size s w ; In the above formula, round() means rounding the number to the nearest integer. S31127: According to A 1_p Height H, Output Height H 2_p Vertical pooling window size k h and vertical step size s h Calculate the vertical padding amount. h According to A 1_p Width W, Output Width W 2_p Horizontal pooling window size k w and horizontal step size s w Calculate the horizontal padding amount w ; Next, for A 1_p Perform zero-fill on all four sides sequentially: first in A 1_p Add padding to the left w Fill the column with zero, then add padding on the right. w Fill the column with zero, then add padding at the top. h Use zero padding in the rows, then add padding at the bottom. h Row zero padding; ultimately resulting in a size of (H + 2 * padding) h )×(W+2*padding w The feature map of the four sides with zero fill; Finally, based on the calculated pooling window size k h k w and step size s h s w Max pooling is performed on the feature map with zero padding on all four sides to obtain a single-channel feature map A of size H / 2×W / 2. 2_p ; The specific steps of S3116 are as follows: S31161: In the first branch, no operation is performed on B1; in the second branch, B1 is first subjected to global average pooling to obtain local attention feature map B2; a fully connected layer is used to reduce the dimensionality of B2 to obtain local attention feature map B3; then B3 is passed through the ReLU activation function to obtain local attention feature map B4; then another fully connected layer is used to increase the dimensionality of B4 to obtain local attention feature map B5, making the number of channels of B5 the same as that of B2; then B5 is passed through the Sigmoid activation function to obtain local attention feature map B6; then B6 is multiplied element-wise with B1 to obtain local attention feature map B7; subsequently, B7 is first subjected to global average pooling, then a multilayer perceptron is used to transform the pooling result, and the transformed result is passed through the Sigmoid activation function to obtain local attention feature map B8; finally, B7 and B8 are multiplied element-wise to obtain local attention feature map B9. B7=B1⊙S(FC2(ReLU(FC1(GAP(B1))))) B9 = B7⊙S(MLP(GAP(B7))) In the above formula, S represents the Sigmoid activation function; GAP() represents the Global Avg Pool operation; ReLU represents the ReLU activation function; MLP() represents multilayer perceptron processing; FC1() represents dimensionality reduction processing through a fully connected layer; FC2() represents dimensionality increase processing through a fully connected layer; ⊙ represents element-wise multiplication operation. S31162: In the third branch, the weight matrix W v Multiplying by B1 yields the value matrix V; in the fourth branch, the weight matrix W is... q Multiplying by B1 yields the query matrix Q; in the fifth branch, the weight matrix W is... k Multiplying Q by B1 yields the key matrix K; then, the dot product of Q and K is calculated, and the attention score matrix S is obtained by applying the Softmax activation function; finally, S is multiplied by V to obtain the global attention feature map B. 10 Next, a 1×1 convolution kernel is applied to B. 10 Perform a convolution operation to obtain the global attention feature map B. 11 Then, for B 11 Global average pooling is performed, followed by feature transformation of the pooling result using a multilayer perceptron. The transformed feature result is then passed through a sigmoid activation function to obtain the global attention feature map B. 12 Subsequently, B 11 With B 12 Perform element-wise multiplication to obtain the global attention feature map B. 13 ; V=W v ×B1 Q=W q ×B1 K=W k ×B1 B 12 =S(MLP(GAP(B 11 ))) B 13 =B 11 ⊙B 12 In the above formula, S represents the Sigmoid activation function; Softmax represents the Softmax activation function; GAP() represents the Global Avg Pool operation; MLP() represents the multilayer perceptron processing; and the He initialization method is used to initialize the weight matrix W. q W k and W v Perform initialization, W q W k and W v All dimensions are Where C ' The number of channels after compression; × indicates matrix multiplication; ⊙ indicates element-wise multiplication; + indicates element-wise addition; S31163: Finally, connect B9 and B 13 Perform element-wise addition to obtain and output the fused attention feature map B. 14 .

5. The power equipment anomaly detection method based on multimodal deep learning according to claim 1, characterized in that, The specific steps of S32 are as follows: S321: Extract the spatial features of the vibration signal from the Gram angular field map in the two-dimensional vibration subset using the Gram angular field map feature extraction network GAFE-Net, and obtain the Gram extracted feature map G. 15 ; S322: Extracting Temporal Features of Vibration Signals: The normalized vibration signals from the one-dimensional vibration subset are input into a bidirectional gated recurrent unit (BiGRU). First, the forward GRU layer in the BiGRU processes the vibration signals from left to right in chronological order, using a gating mechanism containing only update and reset gates to progressively update the forward hidden state and capture the forward temporal pattern of the vibration signals. Simultaneously, the reverse GRU layer processes the signals in reverse order from right to left, controlling the information flow through gating to capture the reverse dependencies of the vibration signals. Then, at each time step, the forward and reverse hidden states are concatenated along the channel dimension to form a fused feature containing bidirectional temporal information. Finally, global average pooling is performed on the fused features of all time steps to obtain the temporal feature map N1. S323: G 15 The vibration feature map of the power equipment vibration signal is obtained by concatenating N1 with N1 along the feature dimension. Then, the vibration feature map of the power equipment vibration signal is classified by an MLP classifier to obtain a two-dimensional vector containing the logits scores of vibration with and without faults. Finally, the two-dimensional vector is normalized by the Softmax activation function to obtain the vibration prediction probability of faults. And vibration prediction fault-free probability The specific steps of S321 are as follows: S3211: Represent any Gram angular field map from the two-dimensional vibration subset as G1, input it into a convolutional layer with a kernel size of 4×4 and a stride of 4, and then perform layer normalization to obtain the Gram extracted feature map G2; then, pass G2 through 3 consecutive Bottleneck modules to obtain the Gram extracted feature map G3; then, pass G3 through a Downsample module to reduce its height by half, width by half, and keep the number of channels unchanged to obtain the Gram extracted feature map G4. Specifically, first, pass G3 through layer normalization, then pass the normalized result through a convolutional layer with a kernel size of 2×2 and a stride of 2 to obtain G4; then, pass G4 through 5 consecutive Bottleneck modules to obtain the Gram extracted feature map G5. S3212: Input G5 into the DAACO module to obtain Gram extracted feature map G6; S3213: Input G6 into the Downsample module to reduce its height and width by half while keeping the number of channels unchanged, to obtain the Gram extraction feature map G7; then, pass G7 through 7 Bottleneck modules to obtain the Gram extraction feature map G8; then, pass G8 through the DAACO module to obtain the Gram extraction feature map G9. S3214: Input G9 into the MLAF module to obtain the Gram extracted feature map G. 10 ; S3215: G 10 The input is fed into the Downsample module, which reduces its height and width by half while keeping the number of channels unchanged, resulting in the Gram extracted feature map G. 11 Then, G 11 The Gram feature map G is obtained by passing through 5 Bottleneck modules consecutively. 12 After that, G 12 The Gram feature map G is obtained through the MLAF module. 13 Next, G 13 Perform global average pooling to obtain the Gram extracted feature map G. 14 Finally, G 14 Perform layer normalization to obtain and output the Gram extracted feature map G. 15 ; The specific steps of S3212 are as follows: S32121: In the first branch, no operation is performed on L1; in the second branch, global average pooling is used to pool L1 to obtain the channel-optimized feature map L2; in the third branch, a pooling window of size 3×3 and a dynamic step size s are first applied. D Averaging is performed on L1, and then global averaging is performed on the result to obtain the channel-optimized feature map L3. In the fourth branch, a pooling window of size 5×5 and a dynamic step size s are first applied. D Perform average pooling on L1, and then perform global average pooling on the result to obtain the channel-optimized feature map L4. L2 = GAP(L1) L3 = GAP(AvgPool3(L1)) L4 = GAP(AvgPool5(L1)) In the above formula, It is a hyperparameter; max() indicates the maximum value calculation operation; GAP() indicates the Global Avg Pool operation; AvgPool3() indicates a pooling window size of 3×3 and a stride of s. D Average pooling operation; AvgPool5() represents a pooling window size of 5×5 with a stride of s. D Average pooling operation; S32122: Adjust the channel sparse parameter W S1 Multiplying by L2 yields the channel-optimized feature map L5; the channel sparsity parameter W is then... S2 Multiplying by L3 yields the channel-optimized feature map L7; the channel sparsity parameter W is then... S3 Multiplying by L4 yields the channel-optimized feature map L9; then, element-wise addition of L5, L7, and L9 yields the channel-optimized feature map L0. 11 Then L 11 After applying the Sigmoid activation function, the channel-optimized feature map L is obtained. 12 Next, L 12 Element-wise multiplication with L1 yields the channel-optimized feature map L. 13 ; L 13 =S(W S1 ×L2+W S2 ×L3+W S3 ×L4)⊙L1 In the above formula, the Xavier initialization method is used to initialize the channel sparse parameter W. S1 W S2 and W S3 Initialize; S represents the Sigmoid activation function; × represents matrix multiplication; ⊙ represents element-wise multiplication; + represents element-wise addition; S32123: Transfer abnormal channel parameter W A1 Multiplying by L2 yields the channel optimization feature map L6; the abnormal channel parameters W are then... A2 Multiplying by L3 yields the channel optimization feature map L8; the abnormal channel parameters W are then... A3 Multiplying by L4 yields the channel-optimized feature map L. 10 Then add L6, L8 and L 10 Perform element-wise addition to obtain the channel-optimized feature map L. 14 Then L 14 After applying the ReLU activation function, the channel-optimized feature map L is obtained. 15 ; L 15 =ReLU(W A1 ×L6+W A2 ×L8+W A3 ×L 10 ) The above formula uses a Gaussian distribution. Initialization method, for abnormal channel parameter W A1 W A2 and W A3 Initialize; ReLU represents the ReLU activation function; × represents matrix multiplication; + represents element-wise addition. S32124: L 13 With L 15 Perform element-wise multiplication to obtain the channel-optimized feature map L. 16 Then, the channel attention parameter W D With L 16 Perform a multiplication operation to obtain the channel-optimized feature map L. 17 Then L 17 After applying the Softmax activation function, the channel-optimized feature map L is obtained. 18 Finally, L 16 With L 18 Perform element-wise multiplication to obtain and output the channel-optimized feature map L. 19 ; L 19 =L 16 ⊙Softmax(W D ×(L 13 ⊙L 15 )) The above formula uses a Gaussian distribution. Initialization method, for channel attention parameter W D Initialize; Softmax represents the Softmax activation function; × represents matrix multiplication; ⊙ represents element-wise multiplication. The specific steps of S3214 are as follows: S32141: In the first branch, no operation is performed on T1; in the second branch, no operation is performed on T1; in the third branch, wavelet transform is first applied to T1 to extract high-frequency anomaly features, resulting in a multi-level fusion feature map T2; then, a 1×1 convolution kernel is applied to T2 to obtain a multi-level fusion feature map T3; finally, T3 is processed through a Sigmoid activation function to obtain a multi-level fusion feature map T4. T4 = S(Conv1(DWT(T1))) In the above formula, DWT() represents the wavelet transform operation; Conv1() represents the convolution operation with a kernel of size 1×1; and S represents the sigmoid activation function. S32142: In the fourth branch, a depthwise separable convolution operation is performed on T1 using a 1×1 convolution kernel to obtain a multi-level fused feature map T5; in the fifth branch, a depthwise separable convolution operation is performed on T1 using a 3×3 convolution kernel to obtain a multi-level fused feature map T6; in the sixth branch, a depthwise separable convolution operation is performed on T1 using a 5×5 convolution kernel to obtain a multi-level fused feature map T7; in the seventh branch, a depthwise separable convolution operation is performed on T1 using a 7×7 convolution kernel to obtain a multi-level fused feature map T8. Next, T5, T6, T7, and T8 are added element-wise to obtain the multi-level fused feature map T9; then, T9 is activated by the Sigmoid activation function to obtain the multi-level fused feature map T. 10 : T 10 =S(DSC1(T1)+DSC3(T1)+DSC5(T1)+DSC7(T1)) In the above formula, DSC1() represents a depthwise separable convolution operation with a 1×1 kernel; DSC3() represents a depthwise separable convolution operation with a 3×3 kernel; DSC5() represents a depthwise separable convolution operation with a 5×5 kernel; DSC7() represents a depthwise separable convolution operation with a 7×7 kernel; + represents element-wise addition; S represents the Sigmoid activation function. S32143: In the eighth branch, average pooling is performed on T1 along the channel dimension. Specifically, it iterates through each spatial position (i,j) of T1 (where i∈[1,H], j∈[1,W]) and calculates the average value of its position across all channels. In the ninth branch, max pooling is performed on T1 along the channel dimension. Specifically, it iterates through each spatial position (i,j) of T1 (where i∈[1,H], j∈[1,W]) and calculates the maximum value of its position across all channels. The results from the eighth and ninth branches are then concatenated along the channel dimension to obtain the multi-level fused feature map T. 11 Subsequently, a 1×1 convolution kernel was applied to T. 11 Perform convolution operations to obtain a multi-level fused feature map T. 12 Then apply a 7×7 convolution kernel to T 12 Perform convolution operations to obtain a multi-level fused feature map T. 13 Using a 3×3 convolution kernel with a dilation rate of 2 on T 12 Perform convolution operations to obtain a multi-level fused feature map T. 14 Then T 13 and T 14 Perform element-wise addition to obtain the multi-level fused feature map T. 15 ; then T 15 After applying the Sigmoid activation function, a multi-level fused feature map T is obtained. 16 : T 15 =S(Conv7(T 12 )+Conv3_2(T 12 )) In the above formula, T1(i,j,k) represents the feature value of T1 at spatial position (i,j) and the k-th channel; max k∈[1,C] () indicates the operation of calculating the maximum value of each spatial location across all channels; Concat() indicates the concatenation operation along the channel dimension; Conv7() indicates the convolution operation with a 7×7 kernel; Conv3_2() indicates the convolution operation with a 3×3 kernel with a dilation of 2; S indicates the Sigmoid activation function; + indicates element-wise addition. S32144: Place T1, T4, T 10 and T 16 Perform element-wise multiplication to obtain the multi-level fused feature map T. 17 Then apply a 1×1 convolution kernel to T. 17 Perform convolution operations to obtain a multi-level fused feature map T. 18 Then T 18 After applying the Softmax activation function, the anomaly mask M is obtained. a ; After that, for M a Perform two sub-branch calculations; in the first sub-branch, calculate M. a The inverse anomaly mask is obtained by using the inverse anomaly mask M. 1-a Then T1 and M 1-a Perform element-wise multiplication to obtain the multi-level fused feature map T. 19 In the second sub-branch, T 17 and M a Perform element-wise multiplication to obtain the multi-level fused feature map T. 20 Finally, T 19 and T 20 Perform element-wise addition to obtain and output the multi-level fused feature map T. 21 ; M a =Softmax(Conv1(T1⊙T4⊙T 10 ⊙T 16 )) M 1-a =1-M a T 21 =T1⊙M 1-a +M a ⊙T 17 In the above formula, Conv1() represents a convolution operation with a kernel of size 1×1; Softmax represents the Softmax activation function; ⊙ represents element-wise multiplication; + represents element-wise addition.

6. The power equipment anomaly detection method based on multimodal deep learning according to claim 1, characterized in that, The specific steps of S33 are as follows: S331: Represent any power equipment image in the image subset as P1, input it into the CBS module to obtain the power equipment detection feature map P2; then, pass P2 through the CBS module to obtain the power equipment detection feature map P3; then, pass P3 through the C2F module to obtain the power equipment detection feature map P4; then, pass P4 through the CBS module to obtain the power equipment detection feature map P5. S332: Input P5 into the CFAE module to obtain the power equipment detection feature map P6; S333: Input P6 into the CBS module to obtain the power equipment detection feature map P7; then, pass P7 through the CFAE module to obtain the power equipment detection feature map P8; next, pass P8 through the CBS module to obtain the power equipment detection feature map P9; finally, pass P9 through the CFAE module to obtain the power equipment detection feature map P... 10 Then P 10 The SPPF module is used to obtain the power equipment detection feature map P. 11 Subsequently, P 11 An upsampling operation is performed to obtain the power equipment detection feature map P. 12 Next, P8 and P 12 By stitching along the channel dimension, the power equipment detection feature map P is obtained. 13 Then P 13 The power equipment detection feature map P is obtained through the C2F module. 14 ; then P 14 An upsampling operation is performed to obtain the power equipment detection feature map P. 15 Next, P6 and P 15 By stitching along the channel dimension, the power equipment detection feature map P is obtained. 16 Then P 16 The power equipment detection feature map P is obtained through the C2F module. 17 ; then P 17 The power equipment detection feature map P is obtained through the CBS module. 18 Next, P 14 and P 18 By stitching along the channel dimension, the power equipment detection feature map P is obtained. 19 Then, P 19 The power equipment detection feature map P is obtained through the C2F module. 20 After that, P 20 The power equipment detection feature map P is obtained through the CBS module. 21 Then P 11 and P 21 By stitching along the channel dimension, the power equipment detection feature map P is obtained. 22 Next, P 22 The power equipment detection feature map P is obtained through the C2F module. 23 ; S334: P 17 The data is input into the AADM module to obtain the power equipment detection feature map P. 24 ; P 20 The data is input into the AADM module to obtain the power equipment detection feature map P. 27 ; P 23 The data is input into the AADM module to obtain the power equipment detection feature map P. 30 ; S335: P 24 The data is input into the AADM module to obtain the power equipment detection feature map P. 25 ; P 27 The data is input into the AADM module to obtain the power equipment detection feature map P. 28 ; P 30 The data is input into the AADM module to obtain the power equipment detection feature map P. 31 ; After that, P 25 The power equipment detection feature map P is obtained by using a convolutional layer with a kernel size of 1×1. 26 ; P 28 The power equipment detection feature map P is obtained by using a convolutional layer with a kernel size of 1×1. 29 ; P 31 The power equipment detection feature map P is obtained by using a convolutional layer with a kernel size of 1×1. 32 ; Subsequently, for P 26 The values ​​at all locations in the spatial dimension are averaged to obtain a two-dimensional vector X1 containing the faulty logits score and the fault-free logits score of the image; for P 29 The values ​​at all locations in the spatial dimension are averaged to obtain a two-dimensional vector X2 containing the faulty logits score and the fault-free logits score of the image; P 32 The values ​​at all locations in the spatial dimension are averaged to obtain a two-dimensional vector X3 containing the image faulty logits score and the image fault-free logits score. Then, the three levels of two-dimensional vectors are summed element-wise to obtain a comprehensive logits vector X4. Finally, the comprehensive logits vector is transformed into a probability distribution through a softmax activation function to obtain a two-dimensional vector P containing the image prediction fault probability and the image prediction fault-free probability. out Then according to P out The probability of failure is obtained from image prediction. And image prediction fault-free probability X4 = X1 + X2 + X3 P out =Softmax(X4) In the above formula, Softmax represents the Softmax activation function; + indicates element-wise addition. The specific steps of S332 are as follows: S3321: Input the power equipment detection feature map V1. First, use the channel shuffling method to cross-mix the channel information of V1 to obtain the channel feature enhancement feature map V2. Then, use segmented GLU to perform gating processing on V2 to obtain the channel feature enhancement feature map V3. The specific steps are: first, divide V2 into two parts along the channel dimension V3. 2_1 and V 2_2 Then V 2_2 The gate signal G is obtained through the Sigmoid activation function, and finally V is... 2_1 Element-wise multiplication with G yields V3; then, V3 is convolved with a 1×1 kernel to obtain the channel feature enhancement feature map V4. S3322: Perform four branch calculations on V4. In the first branch, first, use global average pooling to pool V4 to obtain the channel feature enhancement feature map V5; then, use a 1×1 convolution kernel to convolve V5 to obtain the channel feature enhancement feature map V6; then, pass V6 through the Sigmoid activation function to obtain the channel feature enhancement feature map V7. In the second branch, use a 3×3 convolution kernel to convolve V4 to obtain the channel feature enhancement feature map V8; then, pass V8 through the Tanh activation function to obtain the channel feature enhancement feature map V9. In the third branch, do not perform any operation on V4; then, multiply V7, V9, and V4 element-wise to obtain the channel feature enhancement feature map V. 11 ; S3323: In the fourth branch, V4 is processed by the Bottleneck module for feature extraction to obtain the channel feature enhancement feature map V. 10 Finally, V 11 and V 10 Perform channel-wise maximum fusion to obtain and output channel feature enhancement feature maps V. 12 The specific operation is as follows: For V 11 and V 10 For each corresponding channel, traverse all spatial locations on the feature map of that channel and calculate V. 11 and V 10 The maximum value of the element at that position, and use it as V. 12 Elements in; The specific steps of S334 are as follows: S3341: Input the power equipment detection feature map Y1. First, perform convolution operations on Y1 using a 3×3 kernel, a 5×5 kernel, and a 7×7 kernel. Then, concatenate the three convolution results along the channel dimension to obtain the attention anomaly detection feature map Y2. Next, perform convolution operations on Y2 using a 1×1 kernel to obtain the attention anomaly detection feature map Y3. Then, process Y3 with the ReLU activation function and then process the result with the Hard Sigmoid activation function to obtain the attention anomaly detection feature map Y4. Y2=Concat(Conv3(Y1),Conv5(Y1),Conv7(Y1)) Y4 = HardS(ReLU(Conv1(Y2))) In the above formula, Conv3() represents a convolution operation with a 3×3 kernel; Conv5() represents a convolution operation with a 5×5 kernel; Conv7() represents a convolution operation with a 7×7 kernel; Concat() represents a concatenation operation along the channel dimension; Conv1() represents a convolution operation with a 1×1 kernel; ReLU represents the ReLU activation function; HardS represents the Hard Sigmoid activation function. S3342: Perform element-wise multiplication of Y4 and Y1 to obtain the attention anomaly detection feature map Y5; then apply a depthwise separable convolution operation to Y5 using a 3×3 convolution kernel to obtain the attention anomaly detection feature map Y6; then pass Y6 through the Sigmoid activation function to obtain the attention anomaly detection feature map Y7; finally, perform element-wise multiplication of Y7 and Y5 to obtain the attention anomaly detection feature map Y8; Y8=(Y4⊙Y1)⊙S(DSC3(Y5)) In the above formula, DSC3() represents a depthwise separable convolution operation with a 3×3 kernel; S represents the Sigmoid activation function; ⊙ represents element-wise multiplication. S3343: Use a fully connected layer to perform feature mapping on Y8 to obtain the attention anomaly detection feature map Y9; then, process Y9 through the sigmoid activation function to obtain the attention anomaly detection feature map Yi. 10 Finally, Y 10 Element-wise multiplication with Y8 yields the attention anomaly detection feature map Y. 11 ; Y 11 =S(FC(Y8))⊙Y8 In the above formula, S represents the Sigmoid activation function; FC() represents the feature mapping operation of the fully connected layer; and ⊙ represents the element-wise multiplication operation.

7. The power equipment anomaly detection method based on multimodal deep learning according to claim 1, characterized in that, The specific steps of S34 are as follows: The input sub-model M1 has a fault probability for audio prediction of power equipment. And audio prediction fault-free probability Sub-model M2 has a failure probability for vibration prediction of power equipment. And vibration prediction fault-free probability Sub-model M3 has a fault probability for image prediction of power equipment. And image prediction fault-free probability First, calculate the mutual information I(M1,M2) between M1 and M2, the mutual information I(M1,M3) between M1 and M3, and the mutual information I(M2,M3) between M2 and M3. Then, based on the mutual information calculation results, adjust the weights of each sub-model using a weighted average to obtain the adjusted weight w of model M1. adjust (1) Weights w in the M2 model adjust (2) and the weights w of the M3 model adjust (3); Then apply the weighted voting method to calculate the probability P of a faulty weighted vote. vote (1) and the fault-free weighted voting probability P vote (2); Then, the probability P of a faulty soft vote is calculated using the soft voting method. soft (1) and the probability of fault-free soft voting P soft (2); Finally, the results of the weighted fusion of the weighted voting method and the soft voting method are used to obtain the final prediction result: the overall prediction probability of failure P. mmwf (1) and the overall predicted fault-free probability P mmwf (2); P mmwf (y)=α*P vote (y)+(1-α)*P soft (y) In the above formula, I(M) i M j ) indicates the calculation of M i and M j Mutual information between them; y represents a binary classification value indicating whether there is a fault, where y=1 indicates a fault and y=2 indicates no fault; Submodel M i The predicted probability of category y; P(y) is the prior probability of category y, where P(y=1) equals the ratio of the number of faulty samples in the dataset to the total number of samples, and P(y=2) equals the ratio of the number of fault-free samples in the dataset to the total number of samples; w i Each sub-model M represents i Learnable weights; k represents the number of sub-models; w adjust (i) represents the sub-model M adjusted based on mutual information. i Normalized weights; 1(y=y pred,i ) represents the sub-model M i The indicator function for whether the prediction is for category y; α represents a learnable parameter used to balance the effects of weighted voting and soft voting.

8. The power equipment anomaly detection method based on multimodal deep learning according to claim 1, characterized in that, The specific steps of S4 are as follows: S41: Divide the preprocessed multimodal dataset of power equipment in S2 into training set, validation set and test set according to a certain ratio to ensure that the samples in the training set, validation set and test set are independent and do not overlap; during the partitioning process, ensure that the class distribution of each subset is consistent with the overall dataset to avoid the negative impact of data imbalance on model training; S42: Employs a dynamically adjusted hyperparameter scheme: The basic training cycle is set to a certain number of rounds, and a progressive batch expansion strategy is implemented, that is, the initial batch size is set to a small value, and the batch size is gradually increased after each training cycle. The learning rate scheduling uses the cyclic cosine annealing algorithm, starting with a low baseline value and setting the cycle length according to training needs, so that the learning rate decreases smoothly during training, effectively improving the model convergence. The optimizer is NAdam, and gradient normalization constraints are set to prevent the gradient from being too large by limiting the L2 norm of parameter updates, thus ensuring the stability of the training process. S43: During the training phase, the input samples in each batch are a quintuple, which includes: normalized audio data from the one-dimensional sound subset, Mel spectrograms from the two-dimensional sound subset, normalized vibration signals from the one-dimensional vibration subset, Gram angular field maps from the two-dimensional vibration subset, and images of power equipment from the image subset. Through the quintuple samples, the model can learn the correlation between different modalities. During feature extraction, a cross-modal contrastive loss function is used to optimize the model, enabling data from different modalities to be effectively aligned in the feature space, thereby improving multimodal learning capabilities. S44: During the validation phase, a multi-dimensional evaluation protocol is adopted, and a full validation is performed every few batches. The validation process includes several test modes: standard data inference, noise interference test, and modality missing test. An adaptive early stopping mechanism is set up: when the coefficient of variation of the main evaluation metric remains below the preset value in several consecutive validations, the training is automatically terminated to ensure the convergence and stability of the model. S45: During the testing phase, the model will undergo several rigorous tests on the test set, including: cross-device generalization evaluation, real-time stress testing, model stability, robustness, and performance evaluation under data noise interference; through testing, it is ensured that the model can maintain high reliability and accuracy in practical applications and meet the actual needs of power equipment anomaly detection.

9. The power equipment anomaly detection method based on multimodal deep learning according to claim 1, characterized in that, The specific steps of S5 are as follows: S51: Input the collected audio data, vibration signals and images of the power equipment into the power equipment anomaly detection model to obtain the overall predicted probability of failure and the overall predicted probability of no failure. S52: Then make a judgment. If the overall predicted probability of failure is greater than the overall predicted probability of no failure, then the output device is faulty; otherwise, the output device is not faulty.

Images