Power equipment anomaly detection method based on multi-modal deep learning

Abstract

The invention belongs to the technical field of power equipment anomaly detection, and particularly relates to a power equipment anomaly detection method based on multi-modal deep learning, which comprises the following specific steps: S1, constructing an original power equipment multi-modal data set covering audio data, vibration signals and images of power equipment; s2, preprocessing the original multi-modal data set of the power equipment to obtain a preprocessed multi-modal data set of the power equipment; s3, constructing a power equipment anomaly detection model based on multi-modal deep learning; s4, dividing the preprocessed multi-modal data set of the power equipment, and then training and verifying an anomaly detection model of the power equipment; and S5, detecting the power equipment by using the trained power equipment anomaly detection model. According to the invention, through combining various characteristics of the audio data, the vibration signal and the electrical equipment image, various abnormal performance of the electrical equipment can be comprehensively captured, and the method is suitable for abnormal detection of the electrical equipment in different operation states.

Description

Technical Field

[0001] The present invention relates to the technical field of power equipment anomaly detection, and specifically to a power equipment anomaly detection method based on multimodal deep learning. Background Art

[0002] During long-term operation, power equipment may experience anomalies due to aging, damage, or external factors. Promptly detecting and accurately diagnosing these anomalies is crucial for preventing failures, improving equipment reliability, and ensuring power supply security. Traditional detection methods typically rely on manual inspections or sensor-based data monitoring, but these methods face challenges: manual inspections are inefficient and lack comprehensive coverage; traditional sensors lack sufficient multi-dimensional information, making it difficult to detect potential problems in a timely manner; and manual inspections are subject to subjective factors and cannot provide long-term, real-time monitoring.

[0003] In recent years, the development of artificial intelligence (AI), particularly breakthroughs in deep learning, has ushered in new opportunities for anomaly detection in power equipment. Deep learning models, particularly convolutional neural networks (CNNs) and recurrent neural networks (RNNs), have demonstrated powerful capabilities in complex pattern recognition and feature extraction. By training deep learning models, they can automatically detect potential anomalies and accurately predict equipment status based on historical data and real-time sensor data. This data-driven approach is more efficient and accurate than traditional manual detection, providing real-time early warning information and enhancing the intelligent maintenance of power equipment.

[0004] However, existing methods still face challenges. The operating status of power equipment is affected by multiple factors, and abnormal signals can be weak and similar to normal conditions, making them easily overlooked. Furthermore, power equipment data is multimodal, often requiring the fusion of information from different sensors and data sources to comprehensively assess equipment status. Existing deep learning models can suffer from poor accuracy when processing multimodal data, particularly when noisy, missing, or unbalanced data is present.

[0005] Therefore, improving the accuracy and reliability of power equipment anomaly detection, particularly when processing complex multimodal data, has become a research hotspot. Multimodal deep learning-based methods can integrate diverse data sources to identify potential anomalies, maintaining high accuracy and robustness even in the presence of incomplete or noisy data. By optimizing model structures and training strategies, these methods can effectively enhance the automation and intelligence of detection, providing strong support for the safe operation of power systems.

[0006] However, the existing power equipment anomaly detection model based on multimodal deep learning has the following problems:

[0007] 1. Unable to fully capture the diverse abnormal behaviors of power equipment: Power equipment may experience different types of faults during operation (such as overload, short circuit, mechanical damage, etc.). The abnormal manifestations of these faults are diverse and vary in time and space. Existing models fail to effectively identify all abnormal situations, resulting in insufficient comprehensiveness and accuracy in detection.

[0008] 2. Unable to accurately extract abnormal features from audio data, vibration signals, or images of power equipment: Existing models are unable to fully exploit potential abnormal features when processing data of different modalities, resulting in low accuracy in identifying equipment faults.

[0009] 3. Information imbalance between modalities: Failure to reasonably and effectively utilize the information of each model and assign appropriate weights to the models can easily lead to bias towards a certain modality, affecting the stability and accuracy of the final detection results.

[0010] Based on the above, a method for detecting abnormalities in power equipment based on multimodal deep learning is invented. Summary of the Invention

[0011] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions:

[0012] The power equipment anomaly detection method based on multimodal deep learning includes the following specific steps:

[0013] S1: Build an original multimodal dataset of power equipment covering audio data, vibration signals and images of power equipment;

[0014] S2: preprocessing the original power equipment multimodal dataset to obtain a preprocessed power equipment multimodal dataset;

[0015] S3: Build a power equipment anomaly detection model based on multimodal deep learning;

[0016] S4: Divide the preprocessed power equipment multimodal dataset, and then train and verify the power equipment anomaly detection model;

[0017] S5: Use the trained power equipment anomaly detection model to detect power equipment.

[0018] As a preferred solution of the method for detecting abnormalities in power equipment based on multimodal deep learning according to the present invention, the specific steps of S1 are as follows:

[0019] S11: First, use a sound sensor to collect audio data generated by the power equipment during operation and save it as an audio file; then, use a vibration sensor to collect vibration signals of the power equipment and save it as a vibration data file; then, use a high-definition camera to capture the working area of the power equipment, record images of the power equipment covering key components, and save them as image files;

[0020] S12: Label the collected audio files, vibration data files and image files; determine whether the current equipment has a fault from the audio data in the audio file, and then label the audio data as faulty or not; determine whether the current equipment has a fault from the vibration signal in the vibration data file, and then label the vibration signal as faulty or not; determine whether the current equipment has a fault from the power equipment image in the image file, and then use the image labeling tool to label the power equipment image as faulty or not.

[0021] As a preferred solution of the method for detecting abnormalities in power equipment based on multimodal deep learning according to the present invention, the specific steps of S2 are as follows:

[0022] S21: normalizing the audio data in the audio file, that is, unifying the sampling frequency and sampling duration so that the audio data in the audio file have the same sampling interval, thereby obtaining a one-dimensional sound sub-dataset;

[0023] S22: Processing the audio data in the audio file into a Mel-frequency spectrum using Mel-frequency cepstral coefficients to obtain a two-dimensional sound sub-dataset;

[0024] S23: normalizing the vibration signal in the vibration data file so that its amplitude is within a uniform range, and then dividing the vibration data into multiple time periods using a sliding window method, where each time period contains vibration data of uniform length, to obtain a one-dimensional vibration sub-dataset;

[0025] S24: converting the vibration signal in the vibration data file into a Gram angle field diagram to obtain a two-dimensional vibration sub-dataset;

[0026] S25: resizing the power equipment images in the image file to a uniform size, then using data enhancement technology to generate multiple enhanced images and adding them to the dataset to obtain an image sub-dataset;

[0027] S26: Perform time alignment operation: For each sampling time point, extract data samples at the corresponding moment from all sub-datasets, including the normalized audio data in the one-dimensional sound sub-dataset, the Mel spectrum map in the two-dimensional sound sub-dataset, the normalized vibration signal in the one-dimensional vibration sub-dataset, the Gram angular field map in the two-dimensional vibration sub-dataset, and the power equipment image in the image sub-dataset, and combine them to form a five-tuple sample; the process achieves cross-modal synchronization through timestamp matching to ensure that the data samples of all sub-datasets are aligned in the time dimension; repeat this alignment operation for all available time points to obtain the preprocessed power equipment multimodal dataset.

[0028] As a preferred solution of the method for detecting abnormalities in power equipment based on multimodal deep learning according to the present invention, the specific steps of S3 are as follows:

[0029] S31, anomaly detection based on power equipment audio data: First, the Mel-spectrogram in the two-dimensional sound sub-dataset of S2 is used to extract the frequency domain features of the audio data through MSFE-Net; and the normalized audio data in the one-dimensional sound sub-dataset of S2 is used to extract the time domain features of the audio data through BiLSTM. The two results are then fused at the feature layer through time-frequency collaborative features to obtain the audio feature map of the power equipment audio data. The audio feature map of the power equipment audio data is then classified using the MLP classifier. Finally, the classification results are converted into the audio prediction fault probability and the audio prediction non-fault probability using the Softmax activation function.

[0030] S32, anomaly detection based on vibration signals of power equipment: First, the Gram angle field map in the two-dimensional vibration sub-dataset of S2 is used to extract the spatial features of the vibration signal through GAFE-Net; and the normalized vibration signal in the one-dimensional vibration sub-dataset of S2 is used to extract the time series features of the vibration signal through BiGRU. Subsequently, the two results are combined in the feature layer to perform a time series joint feature fusion to obtain the vibration feature map of the power equipment vibration signal. Then, the vibration feature map of the power equipment vibration signal is classified using the MLP classifier. Finally, the classification results are converted into the vibration prediction fault probability and the vibration prediction non-fault probability using the Softmax activation function.

[0031] S33, anomaly detection based on power equipment images: For power equipment images in image files, ADEI-Net is used to perform target detection on the power equipment images, extract visual features of the equipment, and determine the anomalies of the images. The predicted probability of the images being faulty and the predicted probability of being fault-free are output.

[0032] S34, determine whether the power equipment has a fault: combine the predicted probabilities of each mode, and then perform decision-level fusion through a multi-modal weighted fusion algorithm to obtain the overall predicted probability of fault and the overall predicted probability of no fault; and determine whether the power equipment has a fault based on the probability values; if the overall predicted probability of fault is greater than the overall predicted probability of no fault, then output that the equipment has a fault; otherwise, output that the equipment has no fault.

[0033] As a preferred solution of the method for detecting abnormalities in power equipment based on multimodal deep learning according to the present invention, the specific steps of S31 are as follows:

[0034] S311: The Mel spectrum graph in the two-dimensional sound sub-dataset is passed through the Mel spectrum graph feature extraction network MSFE-Net to extract the frequency domain features of the audio data and obtain the Mel high-level feature graph F 19 ;

[0035] S312: Extracting time domain features of audio data: The normalized audio data in the one-dimensional sound sub-dataset is input into the bidirectional long short-term memory network BiLSTM; first, the forward LSTM layer in the BiLSTM processes the audio data from left to right according to the time step, gradually updating the forward hidden state, thereby capturing the forward dependency of the audio data in the time series; at the same time, the reverse LSTM layer in the BiLSTM processes the audio data from right to left according to the time step, updating the reverse hidden state, thereby capturing the backward dependency of the audio data; next, the forward and reverse hidden states are concatenated at each time step to obtain the bidirectional features of the time step; finally, the bidirectional features of all time steps are globally averaged and pooled to obtain the time domain feature map E1;

[0036] S313: F 19 The audio feature map of the power equipment audio data is obtained by splicing E1 and E2 in the feature dimension. Then, the audio feature map of the power equipment audio data is classified by the MLP classifier to obtain a two-dimensional vector containing the audio fault logits score and the audio fault-free logits score. Finally, the two-dimensional vector is normalized by the Softmax activation function to obtain the audio prediction fault probability. and audio prediction failure-free probability

[0037] The specific steps of S311 are as follows:

[0038] S3111: Denote any mel-spectrogram in the two-dimensional sound sub-dataset as F1, input it into a convolutional layer with 64 3×3 convolution kernels and a stride of 1, and then pass it through the ReLU activation function and batch normalization to obtain the mel shallow feature map F2; then, input F2 into a convolutional layer with 64 3×3 convolution kernels and a stride of 1, and then pass it through the ReLU activation function and batch normalization to obtain the mel shallow feature map F3;

[0039] S3112: Input F3 into the SAPM module to obtain the Mel shallow feature map F4; where F3 is denoted as A1;

[0040] S3113: Input F4 to a convolutional layer with 128 3×3 convolution kernels and a stride of 1, and then pass through the ReLU activation function and batch normalization to obtain the Mel shallow feature map F5; then input F5 to a convolutional layer with 128 3×3 convolution kernels and a stride of 1, and then pass through the ReLU activation function and batch normalization to obtain the Mel shallow feature map F6; then, input F6 to the SAPM module to obtain the Mel shallow feature map F7;

[0041] S3114: Input F7 to a convolutional layer with 256 3×3 convolution kernels and a stride of 1, and then pass through the ReLU activation function and batch normalization to obtain the Mel middle layer feature map F8; then input F8 to a convolutional layer with 256 3×3 convolution kernels and a stride of 1, and then pass through the ReLU activation function and batch normalization to obtain the Mel middle layer feature map F9; then, input F9 to the SAPM module to obtain the Mel middle layer feature map F 10 ;

[0042] S3115: F 10 The input is sent to a convolutional layer with 512 3×3 convolution kernels and a stride of 1, and then processed by ReLU activation function and batch normalization to obtain the Mel high-level feature map F 11 ; Then F 11 The input is sent to a convolutional layer with 512 3×3 convolution kernels and a stride of 1, and then processed by ReLU activation function and batch normalization to obtain the Mel high-level feature map F 12 ;

[0043] S3116: F 12 Input to the LGFA module to obtain the Mel high-level feature map F 13 ;

[0044] S3117: F 13 Input to the SAPM module to obtain the Mel high-level feature map F 14 ; Then F 14The input is sent to a convolutional layer with 512 3×3 convolution kernels and a stride of 1, and then processed by ReLU activation function and batch normalization to obtain the Mel high-level feature map F 15 ; Then, F 15 The input is sent to a convolutional layer with 512 3×3 convolution kernels and a stride of 1, and then processed by ReLU activation function and batch normalization to obtain the Mel high-level feature map F 16 ; Then, F 16 Input to the LGFA module to obtain the Mel high-level feature map F 17 ; Next, F 17 Input to the SAPM module to obtain the Mel high-level feature map F 18 ; Finally, F 18 Input to the fully connected layer, obtain and output Mel high-level feature map F 19 ;

[0045] The specific steps of S3112 are as follows:

[0046] S31121: In the first branch, calculate A along the vertical axis 1_p The mean of the vertical mean vector μ p,h ; In the second branch, calculate A along the vertical axis 1_p The variance of the vertical variance vector

[0047] μ p,h =[μ p,h [1],μ p,h [2],…,μ p,h [h]]

[0048]

[0049] In the above formula, A 1_p (i,j) represents the 1_p The eigenvalue of the (i, j)th position in ;

[0050] S31122: In the third branch, according to A 1_p The output height H is calculated from the height H 2_p ;

[0051]

[0052] S31123: In the fourth branch, A 1_p Do any operation; in the fifth branch, calculate A along the horizontal axis 1_p The mean of the level vector μ p,w , in the sixth branch, calculate A along the horizontal axis 1_p The variance of the horizontal variance vector

[0053] μ p,w =[μ p,w [1],μ p,w [2],…,μ p,w [w]]

[0054]

[0055]

[0056] In the above formula, A 1_p (i,j) represents the 1_p The eigenvalue of the (i, j)th position in ;

[0057] S31124: In the seventh branch, according to A 1_p The width W is calculated to get the output width W 2_p ;

[0058]

[0059] S31125: Based on variance and Calculate and get the variance factor Based on μ p,h Perform variance guided fusion calculation to obtain the mean μ h , then according to Will Perform variance guided fusion calculation to obtain variance Then according to μ p,w Perform variance guided fusion calculation to obtain the mean μ w , then according to Will Perform variance guided fusion calculation to obtain the mean Then, according to μ h and Calculate and get the vertical pooling window size k h ; According to μ w and Calculate and get the horizontal pooling window size k w ;

[0060]

[0061] In the above formula, max() represents the operation of calculating the maximum value; mean() represents the operation of calculating the average value; round() represents the operation of rounding the number to the nearest integer; k minis the size of the minimum pooling window, which is used to avoid the pooling window being too small; ε is a constant used to prevent instability caused by the mean being close to zero or the variance being too small;

[0062] S31126: According to k h and H 2_p Calculate the vertical step length s h ; Then, according to k w and W 2_p Calculate the horizontal step size s w ;

[0063]

[0064] In the above formula, round() means rounding the number to the nearest integer;

[0065] S31127: According to A 1_p Height H, output height H 2_p , vertical pooling window size k h and vertical step length s h , calculate the vertical padding h According to A 1_p Width W, output width W 2_p , horizontal pooling window size k w and horizontal step length s w , calculate the horizontal padding w ;

[0066]

[0067] Next, for A 1_p Perform zero padding on all four sides in sequence: first in A 1_p Add padding to the left w Zero fill the column and then add padding on the right w Zero fill the columns, then add padding on top h Zero padding the rows, then adding padding at the bottom h The row is zero-filled; the final size is (H+2*padding h )×(W+2*padding w )’s four-side zero-filled feature map;

[0068] Finally, according to the calculated pooling window size k h 、k w and step length s h 、s w, perform the maximum pooling operation on the feature map with zero padding on all four sides to obtain a single-channel feature map A of size H / 2×W / 2 2_p ;

[0069] The specific steps of S3116 are as follows:

[0070] S31161: In the first branch, no operation is performed on B1; in the second branch, B1 is first globally average pooled to obtain the local attention feature map B2; a fully connected layer is used to reduce the dimension of B2 to obtain the local attention feature map B3; B3 is then activated by the ReLU function to obtain the local attention feature map B4; another fully connected layer is then used to increase the dimension of B4 to obtain the local attention feature map B5, so that the number of channels of B5 and B2 are consistent; B5 is then activated by the Sigmoid function to obtain the local attention feature map B6; B6 is then element-wise multiplied by B1 to obtain the local attention feature map B7; Subsequently, B7 is first globally average pooled, and the pooling result is transformed by the multi-layer perceptron, and the feature transformation result is passed through the Sigmoid activation function to obtain the local attention feature map B8; Finally, B7 and B8 are element-wise multiplied to obtain the local attention feature map B9;

[0071] B7=B1⊙S(FC2(ReLU(FC1(GAP(B1)))))

[0072] B9=B7⊙S(MLP(GAP(B7)))

[0073] In the above formula, S represents the Sigmoid activation function; GAP() represents the GlobalAvg Pool operation; ReLU represents the ReLU activation function; MLP() represents the multi-layer perceptron processing; FC1() represents the dimensionality reduction processing through the fully connected layer; FC2() represents the dimensionality increase processing through the fully connected layer; ⊙ represents the element-by-element multiplication operation;

[0074] S31162: In the third branch, the weight matrix W v Multiply it with B1 to get the value matrix V; in the fourth branch, the weight matrix W q Multiply it with B1 to get the query matrix Q; in the fifth branch, the weight matrix W k Multiply it with B1 to get the key matrix K; then calculate the dot product of Q and K and pass it through the Softmax activation function to get the attention score matrix S; then multiply S with V to get the global attention feature map B 10 ; Then use the convolution kernel of size 1×1 to B 10 Perform convolution operation to obtain the global attention feature map B 11 ; Then, for B 11Perform global average pooling, then use a multi-layer perceptron to transform the pooling result, and then pass the feature transformation result through the Sigmoid activation function to obtain the global attention feature map B 12 ; Then, B 11 With B 12 Perform element-by-element multiplication to obtain the global attention feature map B 13 ;

[0075] V=W v ×B1

[0076] Q=W q ×B1

[0077] K=W k ×B1

[0078]

[0079] B 12 =S(MLP(GAP(B 11 )))

[0080] B 13 =B 11 ⊙B 12

[0081] In the above formula, S represents the Sigmoid activation function; Softmax represents the Softmax activation function; GAP() represents the Global Avg Pool operation; MLP() represents the multi-layer perceptron processing; the He initialization method is used to initialize the weight matrix W. q 、W k and W v Initialize, W q 、W k and W v The dimensions of Where C' is the number of channels after compression; × represents matrix multiplication; ⊙ represents element-by-element multiplication operation; + represents element-by-element addition operation;

[0082] S31163: Finally, B9 and B 13 Perform element-by-element addition to obtain and output the fused attention feature map B 14 .

[0083] As a preferred solution of the method for detecting abnormalities in power equipment based on multimodal deep learning according to the present invention, the specific steps of S32 are as follows:

[0084] S321: The Gram angular field map in the two-dimensional vibration sub-dataset is passed through the Gram angular field map feature extraction network GAFE-Net to extract the spatial features of the vibration signal and obtain the Gram extraction feature map G15 ;

[0085] S322: Extracting the temporal features of the vibration signal: The normalized vibration signal in the one-dimensional vibration sub-dataset is input into a bidirectional gated recurrent unit (BiGRU). First, the forward GRU layer in the BiGRU processes the vibration signal from left to right in chronological order, and gradually updates the forward hidden state using a gating mechanism consisting only of an update gate and a reset gate to capture the forward temporal pattern of the vibration signal. Simultaneously, the reverse GRU layer processes the signal inversely from right to left, controlling the flow of information through gating to capture the reverse dependency of the vibration signal. Then, at each time step, the forward and reverse hidden states are concatenated according to the channel dimension to form a fused feature containing bidirectional temporal information. Finally, the fused features of all time steps are globally averaged and pooled to obtain a temporal feature map N1.

[0086] S323: G 15 and N1 in the feature dimension to obtain the vibration feature map of the power equipment vibration signal; then, the vibration feature map of the power equipment vibration signal is classified by the MLP classifier to obtain a two-dimensional vector containing the vibration fault logits score and the vibration fault-free logits score; finally, the two-dimensional vector is normalized by the Softmax activation function to obtain the vibration prediction fault probability and vibration prediction failure-free probability

[0087] The specific steps of S321 are as follows:

[0088] S3211: Denote any Gram angular field map in the two-dimensional vibrator dataset as G1, input it into a convolution layer with a convolution kernel size of 4×4 and a stride of 4, and then perform layer normalization to obtain a Gram extraction feature map G2; then, pass G2 through three Bottleneck modules in succession to obtain a Gram extraction feature map G3; then, pass G3 through a Downsample module to reduce its height and width by half while keeping the number of channels unchanged, to obtain a Gram extraction feature map G4. The specific operation is as follows: first, pass G3 through layer normalization, and then pass the layer normalization result through a convolution layer with a convolution kernel size of 2×2 and a stride of 2 to obtain G4; then, pass G4 through five Bottleneck modules in succession to obtain a Gram extraction feature map G5;

[0089] S3212: Input G5 into the DAACO module to obtain the Gram extraction feature map G6;

[0090] S3213: Input G6 into the Downsample module to reduce its height and width by half while keeping the number of channels unchanged, thereby obtaining the Grammar extraction feature map G7. Then, G7 is passed through seven Bottleneck modules in succession to obtain the Grammar extraction feature map G8. Finally, G8 is passed through the DAACO module to obtain the Grammar extraction feature map G9.

[0091] S3214: Input G9 into the MLAF module to obtain the Gram extraction feature map G 10 ;

[0092] S3215: G 10 Input it to the Downsample module to reduce its height and width by half, while keeping the number of channels unchanged, and obtain the Gram extraction feature map G 11 ; Then, G 11 Continuously pass through 5 Bottleneck modules to obtain the Gram extraction feature map G 12 ; Then, G 12 Through the MLAF module, we get the Gram extraction feature map G 13 ; Then, G 13 Perform global average pooling to obtain the Gram extraction feature map G 14 ; Finally, G 14 Perform layer normalization to obtain and output the Gram extraction feature map G 15 ;

[0093] The specific steps of S3212 are as follows:

[0094] S32121: In the first branch, no operation is performed on L1; in the second branch, global average pooling is used to pool L1 to obtain the channel optimized feature map L2; in the third branch, a pooling window of size 3×3 and a dynamic step size s are first used. D Perform an average pooling operation on L1, and then perform a global average pooling operation on the average pooling result to obtain the channel optimized feature map L3; in the fourth branch, first use a pooling window of size 5×5 and a dynamic step size s D Perform an average pooling operation on L1, and then perform a global average pooling operation on the average pooling result to obtain the channel optimized feature map L4;

[0095]

[0096] L2=GAP(L1)

[0097] L3=GAP(AvgPool3(L1))

[0098] L4=GAP(AvgPool5(L1))

[0099] In the above formula, It is a hyperparameter; max() indicates the maximum value operation; GAP() indicates the Global AvgPool operation; AvgPool3() indicates that the pooling window size is 3×3 and the step size is s D Average pooling operation; AvgPool5() means the pooling window size is 5×5 and the step size is s D Average pooling operation;

[0100] S32122: Set the channel sparse parameter W S1 Multiply by L2 to get the channel optimized feature map L5; the channel sparse parameter W S2 Multiply it with L3 to get the channel optimized feature map L7; set the channel sparse parameter W S3 Multiply it with L4 to get the channel optimized feature map L9; then perform element-by-element addition operation on L5, L7 and L9 to get the channel optimized feature map L 11 ; Then L 11 After the Sigmoid activation function, the channel optimized feature map L is obtained 12 ; Then L 12 Perform element-by-element multiplication with L1 to obtain the channel optimized feature map L 13 ;

[0101] L 13 =S(W S1 ×L2+W S2 ×L3+W S3 ×L4)⊙L1

[0102] In the above formula, the Xavier initialization method is used to initialize the channel sparse parameter W S1 、W S2 and W S3 Initialize; S represents the Sigmoid activation function; × represents matrix multiplication; ⊙ represents element-by-element multiplication operation; + represents element-by-element addition operation;

[0103] S32123: Set the abnormal channel parameter W A1 Multiply by L2 to get the channel optimization feature map L6; the abnormal channel parameter W A2 Multiply it by L3 to get the channel optimization feature map L8; the abnormal channel parameter W A3 Multiply by L4 to get the channel optimized feature map L 10 ; Then L6, L8 and L 10 Perform element-by-element addition to obtain the channel optimized feature map L 14 ; Then L 14 After the ReLU activation function, the channel optimized feature map L is obtained 15;

[0104] L 15 =ReLU(W A1 ×L6+W A2 ×L8+W A3 ×L 10 )

[0105] In the above formula, Gaussian distribution is used Initialization method, abnormal channel parameter W A1 、W A2 and W A3 Initialize; ReLU represents the ReLU activation function; × represents matrix multiplication; + represents element-by-element addition operation;

[0106] S32124: L 13 With L 15 Perform element-by-element multiplication to obtain the channel optimized feature map L 16 ; Afterwards, the channel attention parameter W D With L 16 Perform multiplication operation to obtain the channel optimized feature map L 17 ; Then L 17 After the Softmax activation function, the channel optimized feature map L is obtained 18 ; Finally, L 16 With L 18 Perform element-by-element multiplication to obtain and output the channel optimized feature map L 19 ;

[0107] L 19 =L 16 ⊙Softmax(W D ×(L 13 ⊙L 15 ))

[0108] In the above formula, Gaussian distribution is used Initialization method, channel attention parameter W D Initialize; Softmax represents the Softmax activation function; × represents matrix multiplication; ⊙ represents element-by-element multiplication operation;

[0109] The specific steps of S3214 are as follows:

[0110] S32141: In the first branch, no operation is performed on T1; in the second branch, no operation is performed on T1; in the third branch, wavelet transform is first performed on T1 to extract high-frequency abnormal features, and a multi-level fusion feature map T2 is obtained; then a convolution operation is performed on T2 with a convolution kernel of size 1×1 to obtain a multi-level fusion feature map T3; then T3 is activated by the Sigmoid function to obtain a multi-level fusion feature map T4:

[0111] T4 = S(Conv1(DWT(T1)))

[0112] In the above formula, DWT() represents the wavelet transform operation; Conv1() represents the convolution operation of the convolution kernel with a size of 1×1; S represents the Sigmoid activation function;

[0113] S32142: In the fourth branch, a depth-separable convolution operation is performed on T1 using a convolution kernel of size 1×1 to obtain a multi-level fusion feature map T5; in the fifth branch, a depth-separable convolution operation is performed on T1 using a convolution kernel of size 3×3 to obtain a multi-level fusion feature map T6; in the sixth branch, a depth-separable convolution operation is performed on T1 using a convolution kernel of size 5×5 to obtain a multi-level fusion feature map T7; in the seventh branch, a depth-separable convolution operation is performed on T1 using a convolution kernel of size 7×7 to obtain a multi-level fusion feature map T8;

[0114] After that, T5, T6, T7 and T8 are added element by element to obtain the multi-level fusion feature map T9; then T9 is activated by the Sigmoid function to obtain the multi-level fusion feature map T 10 :

[0115] T 10 =S(DSC1(T1)+DSC3(T1)+DSC5(T1)+DSC7(T1))

[0116] In the above formula, DSC1() represents the depth-separable convolution operation of the convolution kernel of size 1×1; DSC3() represents the depth-separable convolution operation of the convolution kernel of size 3×3; DSC5() represents the depth-separable convolution operation of the convolution kernel of size 5×5; DSC7() represents the depth-separable convolution operation of the convolution kernel of size 7×7; + represents the element-by-element addition operation; S represents the Sigmoid activation function;

[0117] S32143: In the eighth branch, an average pooling operation is performed on T1 in the channel dimension. The specific operation is: traverse each spatial position (i, j) of T1 (where i∈[1,H], j∈[1,W]), and calculate the average value of its position in all channels; in the ninth branch, a maximum pooling operation is performed on T1 in the channel dimension. The specific operation is: traverse each spatial position (i, j) of T1 (where i∈[1,H], j∈[1,W]), and calculate the maximum value of its position in all channels; then the results of the eighth and ninth branches are spliced in the channel dimension to obtain a multi-level fusion feature map T 11 ; Then, a convolution kernel of size 1×1 is applied to T 11 Perform convolution operation to obtain a multi-level fusion feature map T 12 ; Then use the convolution kernel of size 7×7 to T 12 Perform convolution operation to obtain a multi-level fusion feature map T 13 ; Use a convolution kernel of size 3×3 and a dilation rate of 2 to T 12 Perform convolution operation to obtain a multi-level fusion feature map T 14 ; Then T 13 and T 14 Perform element-by-element addition to obtain a multi-level fusion feature map T 15 ; Then T 15 After the Sigmoid activation function, the multi-level fusion feature map T is obtained 16 :

[0118]

[0119] T 15 =S(Conv7(T 12 )+Conv3_2(T 12 ))

[0120] In the above formula, T1(i,j,k) represents the eigenvalue of T1 at the spatial position (i,j) and the kth channel; max k∈[1,C] () indicates the operation of calculating the maximum value of each spatial position in all channels; Concat() indicates the concatenation operation in the channel dimension; Conv7() indicates the convolution operation of a convolution kernel of size 7×7; Conv3_2() indicates the convolution operation of a convolution kernel of size 3×3 and a void rate of 2; S indicates the Sigmoid activation function; + indicates the element-by-element addition operation;

[0121] S32144: T1, T4, T 10 and T 16 Perform element-by-element multiplication to obtain a multi-level fusion feature map T 17 ; Then use the convolution kernel of size 1×1 to T17 Perform convolution operation to obtain a multi-level fusion feature map T 18 ; Then T 18 After the Softmax activation function, the abnormal mask M is obtained a ;

[0122] Afterwards, M a Perform two small branch calculations; in the first small branch, calculate M a The reverse exception mask of 1-a ; Then T1 and M 1-a Perform element-by-element multiplication to obtain a multi-level fusion feature map T 19 ; In the second branch, T 17 and M a Perform element-by-element multiplication to obtain a multi-level fusion feature map T 20 ; Finally, T 19 and T 20 Perform element-by-element addition to obtain and output a multi-level fusion feature map T 21 ;

[0123] M a =Softmax(Conv1(T1⊙T4⊙T 10 ⊙T 16 ))

[0124] M 1-a =1-M a

[0125] T 21 =T1⊙M 1-a +M a ⊙T 17

[0126] In the above formula, Conv1() represents the convolution operation of the convolution kernel with a size of 1×1; Softmax represents the Softmax activation function; ⊙ represents the element-by-element multiplication operation; + represents the element-by-element addition operation.

[0127] As a preferred solution of the method for detecting abnormalities in power equipment based on multimodal deep learning according to the present invention, the specific steps of S33 are as follows:

[0128] S331: Denote any power equipment image in the image sub-dataset as P1 and input it into the CBS module to obtain a power equipment detection feature map P2; then, pass P2 through the CBS module to obtain a power equipment detection feature map P3; then, pass P3 through the C2F module to obtain a power equipment detection feature map P4; then, pass P4 through the CBS module to obtain a power equipment detection feature map P5;

[0129] S332: Input P5 into the CFAE module to obtain the power equipment detection characteristic diagram P6;

[0130] S333: Input P6 into the CBS module to obtain the power equipment detection feature map P7; then, pass P7 through the CFAE module to obtain the power equipment detection feature map P8; then, pass P8 through the CBS module to obtain the power equipment detection feature map P9; then, pass P9 through the CFAE module to obtain the power equipment detection feature map P 10 ; Then P 10 Through the SPPF module, the power equipment detection feature map P is obtained 11 ; Then, P 11 Perform upsampling operation to obtain the power equipment detection feature map P 12 ; Next, P8 and P 12 Splicing is performed on the channel dimension to obtain the power equipment detection feature map P 13 ; Then P 13 Through the C2F module, the power equipment detection feature map P is obtained 14 ; Then P 14 Perform upsampling operation to obtain the power equipment detection feature map P 15 ; Next, P6 and P 15 Splicing is performed on the channel dimension to obtain the power equipment detection feature map P 16 ; Then P 16 Through the C2F module, the power equipment detection feature map P is obtained 17 ; Then P 17 Through the CBS module, the power equipment detection characteristic map P is obtained 18 ; Next, P 14 and P 18 Splicing is performed on the channel dimension to obtain the power equipment detection feature map P 19 ; Then, P 19 Through the C2F module, the power equipment detection feature map P is obtained 20 ; Then, P 20 Through the CBS module, the power equipment detection characteristic map P is obtained 21 ; Then P 11 and P 21 Splicing is performed on the channel dimension to obtain the power equipment detection feature map P 22 ; Next, P 22 Through the C2F module, the power equipment detection feature map P is obtained 23 ;

[0131] S334: P 17 Input to the AADM module to obtain the power equipment detection feature map P 24 ;P20 Input to the AADM module to obtain the power equipment detection feature map P 27 ;P 23 Input to the AADM module to obtain the power equipment detection feature map P 30 ;

[0132] S335: P 24 Input to the AADM module to obtain the power equipment detection feature map P 25 ;P 27 Input to the AADM module to obtain the power equipment detection feature map P 28 ;P 30 Input to the AADM module to obtain the power equipment detection feature map P 31 ;

[0133] Afterwards, P 25 Through the convolution layer with a convolution kernel size of 1×1, the power equipment detection feature map P is obtained 26 ;P 28 Through the convolution layer with a convolution kernel size of 1×1, the power equipment detection feature map P is obtained 29 ;P 31 Through the convolution layer with a convolution kernel size of 1×1, the power equipment detection feature map P is obtained 32 ;

[0134] Then, P 26 The values of all positions in the spatial dimension are averaged to obtain a two-dimensional vector X1 containing the image fault logits score and the image fault-free logits score; 29 The values at all positions in the spatial dimension are averaged to obtain a two-dimensional vector X2 containing the image fault logits score and the image fault-free logits score; P 32 The values at all positions in the spatial dimension are averaged to obtain a two-dimensional vector X3 containing the image fault logits score and the image fault-free logits score; then, the three-level two-dimensional vectors are added element by element to obtain the comprehensive logits vector X4; finally, the comprehensive logits vector is converted into a probability distribution through the Softmax activation function to obtain a two-dimensional vector P containing the image prediction fault probability and the image prediction fault-free probability. out , then according to P out Get the image prediction failure probability And the image prediction failure-free probability

[0135] X4=X1+X2+X3

[0136] P out=Softmax(X4)

[0137]

[0138] In the above formula, Softmax represents the Softmax activation function; + represents the element-by-element addition operation;

[0139] The specific steps of S332 are as follows:

[0140] S3321: Input the power equipment detection feature map V1. First, use the channel shuffling method to perform cross-mixing operations on the channel information of V1 to obtain the channel feature enhancement feature map V2. Then, use the split GLU to perform gate processing on V2 to obtain the channel feature enhancement feature map V3. The specific steps are: first, divide V2 into two parts along the channel dimension V 2_1 and V 2_2 , and then V 2_2 Through the Sigmoid activation function, the gate signal G is obtained, and finally V 2_1 Perform element-wise multiplication with G to obtain V3; then, use a convolution kernel of size 1×1 to perform a convolution operation on V3 to obtain a channel feature enhanced feature map V4;

[0141] S3322: Perform four branch calculations on V4. In the first branch, first use global average pooling to perform a pooling operation on V4 to obtain a channel feature enhanced feature map V5; then use a convolution kernel of size 1×1 to perform a convolution operation on V5 to obtain a channel feature enhanced feature map V6; then V6 is passed through a Sigmoid activation function to obtain a channel feature enhanced feature map V7; in the second branch, use a convolution kernel of size 3×3 to perform a convolution operation on V4 to obtain a channel feature enhanced feature map V8; then V8 is passed through a Tanh activation function to obtain a channel feature enhanced feature map V9; in the third branch, no operation is performed on V4; then, V7, V9 and V4 are element-wise multiplied to obtain a channel feature enhanced feature map V 11 ;

[0142] S3323: In the fourth branch, V4 is extracted through the Bottleneck module to obtain the channel feature enhancement feature map V 10 ; Finally, V 11 and V 10 Perform channel-by-channel maximum fusion to obtain and output the channel feature enhancement feature map V 12 , the specific operation is: for V 11 and V 10 For each corresponding channel, traverse all spatial positions on the channel feature map and calculate V 11 and V 10 The maximum value of the element at that position is used as V12 Elements in

[0143] The specific steps of S334 are as follows:

[0144] S3341: Input the power equipment detection feature map Y1. First, convolve Y1 with a 3×3 convolution kernel, a 5×5 convolution kernel, and a 7×7 convolution kernel. Then, concatenate the three convolution results in the channel dimension to obtain the attention anomaly detection feature map Y2. Then, convolve Y2 with a 1×1 convolution kernel to obtain the attention anomaly detection feature map Y3. Then, first process Y3 with the ReLU activation function, and then process the processed result with the Hard Sigmoid activation function to obtain the attention anomaly detection feature map Y4.

[0145] Y2=Concat(Conv3(Y1),Conv5(Y1),Conv7(Y1))

[0146] Y4=HardS(ReLU(Conv1(Y2)))

[0147] In the above formula, Conv3() represents the convolution operation of the convolution kernel of size 3×3; Conv5() represents the convolution operation of the convolution kernel of size 5×5; Conv7() represents the convolution operation of the convolution kernel of size 7×7; Concat() represents the splicing operation in the channel dimension; Conv1() represents the convolution operation of the convolution kernel of size 1×1; ReLU represents the ReLU activation function; HardS represents the Hard Sigmoid activation function;

[0148] S3342: Perform element-wise multiplication on Y4 and Y1 to obtain an attention anomaly detection feature map Y5; then perform a depthwise separable convolution operation on Y5 using a 3×3 convolution kernel to obtain an attention anomaly detection feature map Y6; then pass Y6 through a sigmoid activation function to obtain an attention anomaly detection feature map Y7; then perform element-wise multiplication on Y7 and Y5 to obtain an attention anomaly detection feature map Y8;

[0149] Y8=(Y4⊙Y1)⊙S(DSC3(Y5))

[0150] In the above formula, DSC3() represents the depth-wise separable convolution operation of the convolution kernel of size 3×3; S represents the Sigmoid activation function; ⊙ represents the element-wise multiplication operation;

[0151] S3343: Use the fully connected layer to perform feature mapping on Y8 to obtain the attention anomaly detection feature map Y9; then, Y9 is processed by the Sigmoid activation function to obtain the attention anomaly detection feature map Y 10 ; Finally, Y 10 Perform element-wise multiplication with Y8 to obtain the attention anomaly detection feature map Y 11 ;

[0152] Y 11 =S(FC(Y8))⊙Y8

[0153] In the above formula, S represents the Sigmoid activation function; FC() represents the fully connected layer feature mapping operation; ⊙ represents the element-by-element multiplication operation.

[0154] As a preferred solution of the method for detecting abnormalities in power equipment based on multimodal deep learning according to the present invention, the specific steps of S34 are as follows:

[0155] Input sub-model M1 predicts the fault probability of the audio frequency of the power equipment and audio prediction failure-free probability Sub-model M2 predicts the probability of failure of power equipment vibration and vibration prediction failure-free probability Sub-model M3 predicts the probability of failure of power equipment images And the image prediction probability of no failure First, calculate the mutual information I(M1,M2) between M1 and M2, calculate the mutual information I(M1,M3) between M1 and M3, and calculate the mutual information I(M2,M3) between M2 and M3. Then, adjust the weight of each sub-model according to the mutual information calculation results to obtain the adjusted M1 model weight w adjust (1) M2 model weight w adjust (2) and M3 model weight w adjust (3) Then apply the weighted voting method to calculate the weighted voting probability P of the fault vote (1) and the weighted voting probability P of no fault vote (2); Then the soft voting method is applied to calculate the soft voting probability P of the faulty soft (1) and the probability of fault-free soft voting P soft (2); Finally, the results of weighted fusion weighted voting method and soft voting method are weighted to obtain the final prediction result: the overall predicted fault probability P mmwf (1) and the overall predicted failure-free probability P mmwf (2);

[0156]

[0157] P mmwf (y)=α*P vote (y)+(1-α)*P soft (y)

[0158] In the above formula, I(M i ,M j ) indicates the calculation of M i and M j The mutual information between them; y represents the binary classification value of whether there is a fault, where y = 1 means there is a fault and y = 2 means there is no fault; Represents the sub-model M i The predicted probability of category y; P(y) is the prior probability of category y, P(y=1) is equal to the ratio of the number of faulty samples to the total number of samples in the data set, and P(y=2) is equal to the ratio of the number of non-faulty samples to the total number of samples in the data set; w i Represents each sub-model M i The learnable weights; k represents the number of sub-models; w adjust (i) represents the sub-model M after adjustment based on mutual information i Normalized weight of 1(y=y pred,i ) represents the sub-model M i An indicator function indicating whether the prediction is for category y; α represents a learnable parameter used to balance the influence of weighted voting and soft voting.

[0159] As a preferred solution of the method for detecting abnormalities in power equipment based on multimodal deep learning according to the present invention, the specific steps of S4 are as follows:

[0160] S41: Divide the pre-processed multimodal dataset of power equipment in S2 into a training set, a validation set, and a test set according to a certain ratio, ensuring that the samples in the training set, validation set, and test set are independent and non-overlapping; during the division process, ensure that the category distribution of each subset is consistent with the overall dataset to avoid the negative impact of data imbalance on model training;

[0161] S42: A dynamically adjusted hyperparameter scheme is adopted: the basic training cycle is set to a certain number of rounds, and a progressive batch expansion strategy is implemented, that is, the initial batch size is set to a small value, and the batch size is gradually increased after each training cycle. The learning rate scheduling adopts the cyclic cosine annealing algorithm, starting from a low baseline value, and setting the cycle length according to the training requirements, so that the learning rate decreases smoothly during the training process, effectively improving the convergence of the model. The optimizer uses NAdam and sets the gradient normalization constraint. By limiting the L2 norm of the parameter update, the gradient is prevented from being too large, ensuring the stability of the training process.

[0162] S43: During the training phase, the input samples in each batch are a five-tuple, consisting of: normalized audio data from the one-dimensional sound sub-dataset, mel-spectrograms from the two-dimensional sound sub-dataset, normalized vibration signals from the one-dimensional vibration sub-dataset, Gram angular field images from the two-dimensional vibration sub-dataset, and images of power equipment from the image sub-dataset. Using these five-tuple samples, the model can learn the correlations between different modalities. During feature extraction, a cross-modal contrast loss function is used to optimize the model, effectively aligning data from different modalities in the feature space and improving multimodal learning capabilities.

[0163] S44: During the validation phase, a multi-dimensional evaluation protocol is used, with full validation performed every several batches. The validation process includes several test modes: standard data inference, noise interference testing, and modal loss testing. An adaptive early stopping mechanism is also implemented: when the coefficient of variation of the main evaluation metric remains below a preset value for several consecutive validation runs, training is automatically terminated to ensure model convergence and stability.

[0164] S45: During the testing phase, the model will undergo several rigorous tests on the test set, including: cross-device generalization evaluation, real-time stress testing, model stability and robustness, and performance evaluation under data noise interference; through testing, it is ensured that the model can maintain high reliability and accuracy in actual applications and meet the actual needs of power equipment anomaly detection.

[0165] As a preferred solution of the method for detecting abnormalities in power equipment based on multimodal deep learning according to the present invention, the specific steps of S5 are as follows:

[0166] S51: Input the collected audio data, vibration signals, and images of the power equipment into a power equipment anomaly detection model to obtain an overall predicted probability of failure and an overall predicted probability of no failure;

[0167] S52: Then a judgment is made. If the overall predicted probability of failure is greater than the overall predicted probability of no failure, the output device is faulty; otherwise, the output device is not faulty.

[0168] Compared with existing technologies:

[0169] By combining the diverse features of audio data, vibration signals, and images of power equipment, the present invention can not only comprehensively capture diverse abnormal performances of power equipment and adapt to abnormal detection of power equipment under different operating conditions; it can also effectively extract abnormal features from audio data, vibration signals, and images of power equipment, ensuring full mining of potential abnormal information in different modal data and overcoming the problem of being unable to accurately identify abnormal features; it also ensures the rational fusion of information from each modality, effectively avoiding the problem of information imbalance between modalities, and improving the stability and accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0170] Figure 1 It is a schematic diagram of the process of the present invention;

[0171] Figure 2 This is the overall architecture diagram of the power equipment anomaly detection model of the present invention;

[0172] Figure 3 This is the overall structure diagram of the Mel-spectrogram feature extraction network MSFE-Net of the present invention;

[0173] Figure 4 This is the overall structure diagram of the spatial adaptive pooling module SAPM of the present invention;

[0174] Figure 5 This is the overall structure diagram of the single-channel spatial adaptive pooling module SSAPM of the present invention;

[0175] Figure 6 This is the overall structure diagram of the local-global fusion attention module LGFA of the present invention;

[0176] Figure 7 This is the overall structure diagram of the Gram angle field feature extraction network GAFE-Net of the present invention;

[0177] Figure 8 This is the overall structure diagram of the dynamic anomaly perception channel optimization module DAACO of the present invention;

[0178] Figure 9 This is the overall structure diagram of the multi-level adaptive fusion module MLAF of the present invention;

[0179] Figure 10 This is the overall structure diagram of the power equipment image anomaly detection network ADEI-Net of the present invention;

[0180] Figure 11 This is the overall structure diagram of the channel feature adaptive enhancement module CFAE of the present invention;

[0181] Figure 12 1 is the overall structure diagram of the attention anomaly detection module AADM of the present invention;

[0182] Figure 13 This is the overall structure diagram of the multimodal weighted fusion algorithm MMWF of the present invention. DETAILED DESCRIPTION

[0183] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0184] The present invention provides a method for detecting abnormalities in power equipment based on multimodal deep learning. Figures 1-13 , including the following specific steps:

[0185] S1: Build an original multimodal dataset of power equipment covering audio data, vibration signals and images of power equipment;

[0186] The specific steps of S1 are as follows:

[0187] S11: First, use a sound sensor to collect audio data generated by the power equipment during operation and save it as an audio file; then, use a vibration sensor (including but not limited to an accelerometer, vibration sensor, etc.) to collect vibration signals of the power equipment and save them as a vibration data file; then, use a high-definition camera (including but not limited to a fixed-position camera or a drone, etc.) to photograph the working area of the power equipment, record images of the power equipment covering key components, and save them as image files (including but not limited to JPEG or PNG formats, etc.);

[0188] S12: Labeling the collected audio files, vibration data files, and image files; judging whether the current device has a fault based on the audio data in the audio files, and then labeling the audio data as having a fault or not having a fault; judging whether the current device has a fault based on the vibration signal in the vibration data files, and then labeling the vibration signal as having a fault or not having a fault; judging whether the current device has a fault based on the image of the power equipment in the image files, and then labeling the image of the power equipment as having a fault or not having a fault using an image labeling tool;

[0189] S1 embodiment:

[0190] Collect audio files, vibration data files, and image files, and annotate them to obtain the original multimodal dataset of power equipment. The specific steps are as follows:

[0191] First, collect audio files: Use a microphone device to collect audio data during the operation of power equipment. The sampling frequency is set to 16kHz and the recording time is 10 seconds. 2000 audio data points are collected and saved as audio files in the WAV file format.

[0192] Second, collect vibration data files: Use an accelerometer to collect vibration signals from power equipment at a sampling frequency of 100 Hz and a sampling duration of 10 seconds. Collect 2,000 sets of vibration signals and save them as vibration data files in TXT format, with each file containing 1,000 data points.

[0193] Then, collect image files: Use high-definition camera equipment (such as fixed cameras or drones) to shoot the working area of the power equipment to obtain power equipment images; the size of each power equipment image file is 1920×1080. A total of 1000 images are collected and saved as power equipment image files in JPEG format;

[0194] Finally, the following steps were performed: the audio data were manually labeled and marked as “faulty” and “no fault”, with a total of 1,000 “faulty” data and 1,000 “no fault” data; the vibration signals were manually labeled and marked as “faulty” and “no fault”, with a total of 1,000 “faulty” data and 1,000 “no fault” data; the power equipment images were labeled and an image annotation tool was used to classify each image and mark it as “faulty” and “no fault”, with a total of 500 “faulty” images and 500 “no fault” images.

[0195] S2: preprocessing the original power equipment multimodal dataset to obtain a preprocessed power equipment multimodal dataset;

[0196] The specific steps of S2 are as follows:

[0197] S21: normalizing the audio data in the audio file, that is, unifying the sampling frequency and sampling duration so that the audio data in the audio file have the same sampling interval, thereby obtaining a one-dimensional sound sub-dataset;

[0198] S22: The audio data in the audio file is processed into a Mel-frequency spectrum using Mel-frequency cepstral coefficients to obtain a two-dimensional sound sub-dataset; the specific operation of the Mel-frequency cepstral coefficients is as follows: first, a pre-emphasis operation is performed on the audio data to enhance the high-frequency components and weaken the influence of the low-frequency components; then, the audio data is framed; then, each frame of the audio signal in the audio data is windowed using a Hamming window; then, a short-time Fourier transform (STFT) is performed on the windowed audio signal to obtain a spectrum; then, the spectrum is converted into a Mel-frequency feature through a Mel-frequency filter bank; then, a logarithmic operation is performed on the Mel-frequency feature to obtain a Mel-frequency spectrum; finally, a discrete cosine transform (DCT) is performed on the Mel-frequency spectrum to extract the Mel-frequency spectrum;

[0199] Mel-Frequency Cepstral Coefficients (MFCC) are an audio feature extraction method designed based on the human hearing characteristics and are widely used in fields such as speech recognition. Its core idea is to use the Mel-frequency scale to simulate the nonlinear perception of the human ear to sounds of different frequencies, as the human ear is more sensitive to changes in low-frequency sounds than high-frequency sounds. MFCC can effectively capture important features in audio signals, such as speech formants and fundamental frequency information, while ignoring some details that are not important for recognition.

[0200] Mel spectrogram is a time-frequency representation method based on the Mel scale. It converts the signal spectrum into a nonlinear scale that conforms to the human auditory characteristics, thereby more effectively characterizing the perceptual characteristics of sound. Unlike ordinary spectrograms, Mel spectrograms use a Mel filter bank to divide the spectrum non-uniformly, with higher resolution in the low-frequency part and lower resolution in the high-frequency part, which is closer to the nonlinear perception of pitch by the human auditory system. This representation method can highlight key frequency band information (such as formants and timbre) in signals such as speech, while reducing high-frequency redundancy.

[0201] The Hamming window is a commonly used window function in signal processing, primarily used to reduce spectrum leakage during frequency domain analysis. When performing a discrete Fourier transform (DFT) on a signal, if the signal is not exactly periodic, undesirable sidelobes will appear in its spectrum, leading to spectrum leakage. The Hamming window effectively reduces this spectrum leakage by applying smooth weights to both ends of the signal, causing the signal to gradually decay to zero at both ends. It is a cosine window, shaped like a bell curve with a high center and low ends, which effectively balances the relationship between the mainlobe width and sidelobe attenuation. In practical applications, the Hamming window is often used in speech signal processing, audio analysis, and radar signal processing to help improve the accuracy and reliability of spectrum analysis, make the spectrum clearer, and facilitate the extraction of signal feature information.

[0202] S23: normalizing the vibration signal in the vibration data file so that its amplitude is within a uniform range, and then dividing the vibration data into multiple time periods using a sliding window method, where each time period contains vibration data of uniform length, to obtain a one-dimensional vibration sub-dataset;

[0203] S24: Convert the vibration signal in the vibration data file into a Gram angular field diagram to obtain a two-dimensional vibration sub-dataset; first, normalize the vibration signal; then, map the amplitude and timestamp of each time point into the radius and angle of the polar coordinate system; then, generate a Gram matrix by calculating the triangular inner product of all time point pairs; finally, perform trigonometric function mapping on the Gram matrix to obtain the Gram angular field diagram;

[0204] S25: Adjust the power equipment images in the image file to a uniform size, and then use data enhancement technology (including but not limited to rotation, translation, cropping, flipping, adjusting brightness and contrast, etc.) to generate multiple enhanced images and add them to the dataset to obtain an image sub-dataset;

[0205] S26: Performing a time alignment operation: For each sampling time point, extracting data samples at the corresponding time from all sub-datasets, including the normalized audio data in the one-dimensional sound sub-dataset, the Mel-spectrogram in the two-dimensional sound sub-dataset, the normalized vibration signal in the one-dimensional vibration sub-dataset, the Gram angular field image in the two-dimensional vibration sub-dataset, and the power equipment image in the image sub-dataset, and combining them to form a five-tuple sample; this process achieves cross-modal synchronization through timestamp matching to ensure that the data samples of all sub-datasets are aligned in the time dimension; repeating this alignment operation for all available time points to obtain a preprocessed power equipment multimodal dataset;

[0206] S2 embodiment:

[0207] The original multimodal dataset of electric power equipment is preprocessed to obtain a preprocessed multimodal dataset of electric power equipment. The specific steps are as follows:

[0208] First, the audio data from the 2,000 audio files were normalized to a unified sampling frequency of 16 kHz and a sampling duration of 10 seconds. This ensured that the sampling interval of the audio data was consistent and the length was 160,000 (16 kHz × 10 seconds), resulting in a one-dimensional sound sub-dataset.

[0209] Secondly, the audio data in the audio file is processed with Mel-frequency cepstral coefficients to obtain a two-dimensional sound sub-dataset. The specific steps are as follows: first, the audio data is pre-emphasized, and the filter coefficient is set to 0.97 to enhance the high-frequency part; second, the audio data is divided into 25 millisecond frames with an overlap of 50% (i.e., 160 samples per frame, 80 samples overlapped); third, a Hamming window is applied to each frame of audio data to obtain a windowed frame signal; fourth, a short-time Fourier transform is performed on each windowed frame signal to obtain a spectrogram; fifth, 40 Mel filters are used to map the spectrogram to the Mel frequency domain; sixth, a logarithmic operation is performed on the Mel frequency domain to obtain a Mel spectrum; seventh, a discrete cosine transform is performed on the Mel spectrum to extract the Mel spectrum; eighth, the obtained Mel spectrum with a size of 40×2000 (40 Mel filters, 2000 frames) is used to form a two-dimensional sound sub-dataset;

[0210] The vibration signals in the 2,000 vibration data files were then normalized so that the data amplitudes were uniformly between 0 and 1. The normalized vibration signals were then divided into multiple time periods using a sliding window method (with a window length of 500 points and a step size of 50 points). Each time period consisted of 500 data points, resulting in 80,000 vibration data periods and a one-dimensional vibration sub-dataset.

[0211] Afterwards, the vibration signals in the vibration data file were converted to obtain a two-dimensional vibration sub-dataset. The specific steps were as follows: first, the vibration signals were normalized; second, the amplitude and timestamp of each time point were mapped to the radius and angle of the polar coordinate system; third, the triangular inner product of all time point pairs was calculated to generate a Gram matrix; fourth, trigonometric function mapping was performed on the Gram matrix to obtain a Gram angular field map; fifth, the obtained Gram angular field map of size 100×100 (100×100 Gram matrix) was combined into a two-dimensional vibration sub-dataset;

[0212] Next, the power equipment images in the 1,000 image files were uniformly resized to 224×224. Data augmentation was then performed on the images, using operations such as rotation (maximum angle 30°), translation (maximum translation 20 pixels), cropping (maximum cropping ratio 0.2), flipping (horizontally flipping), and adjusting brightness and contrast. Multiple enhanced images were generated and added to the dataset, resulting in an image sub-dataset containing 2,000 power equipment images.

[0213] Next, a strict time alignment operation is performed: for each sampling time point, the data samples at the corresponding moment are extracted from all sub-datasets, including the normalized audio data from the one-dimensional sound sub-dataset, the Mel-spectrogram from the two-dimensional sound sub-dataset, the normalized vibration signal from the one-dimensional vibration sub-dataset, the Gram angular field image from the two-dimensional vibration sub-dataset, and the power equipment image from the image sub-dataset, and the combination forms a complete five-tuple sample. This process achieves cross-modal synchronization through timestamp matching, ensuring that the data samples of all sub-datasets are precisely aligned in the time dimension. This alignment operation is repeated for all available time points to obtain the preprocessed power equipment multimodal dataset.

[0214] S3: Build a power equipment anomaly detection model based on multimodal deep learning;

[0215] This paper designs a power equipment anomaly detection model based on multimodal deep learning. Specifically, it includes processing three modal data types: audio data, vibration signals, and power equipment images, feature extraction, and decision-level fusion. Finally, it outputs whether the power equipment is faulty.

[0216] The model consists of three sub-models and a multimodal weighted fusion algorithm: Sub-model M1 is used to detect power equipment anomalies on audio data in audio files. After processing the audio data with time-frequency collaborative features and classifying it, the audio predicted fault probability and the audio predicted non-fault probability of the power equipment are output; Sub-model M2 is used to detect power equipment anomalies on vibration signals in vibration data files. After processing the vibration signals with time-series joint features and classifying them, the vibration predicted fault probability and the vibration predicted non-fault probability of the power equipment are output; Sub-model M3 is used to detect power equipment anomalies on power equipment images in image files. The power equipment images are processed by a network model to output the image predicted fault probability and the image predicted non-fault probability of the power equipment; The multimodal weighted fusion algorithm performs decision-level fusion on the predicted probabilities of each sub-model and outputs the probability of whether the power equipment has a fault (divided into two categories: overall predicted fault probability and overall predicted non-fault probability), thereby realizing a comprehensive evaluation of the operating status of the power equipment; The overall architecture of the power equipment anomaly detection model is as follows: Figure 2 As shown;

[0217] The specific steps of S3 are as follows:

[0218] S31, anomaly detection based on power equipment audio data: First, the Mel-spectrogram in the two-dimensional sound sub-dataset of S2 is used to extract the frequency domain features of the audio data through MSFE-Net; and the normalized audio data in the one-dimensional sound sub-dataset of S2 is used to extract the time domain features of the audio data through BiLSTM. The two results are then fused at the feature layer through time-frequency collaborative features to obtain the audio feature map of the power equipment audio data. The audio feature map of the power equipment audio data is then classified using the MLP classifier. Finally, the classification results are converted into the audio prediction fault probability and the audio prediction non-fault probability using the Softmax activation function.

[0219] The MLP (Multi-layer Perceptron) classifier is a classic feedforward artificial neural network consisting of an input layer, multiple hidden layers, and an output layer, achieving nonlinear classification through a fully connected structure. It uses a backpropagation algorithm to optimize network weights and introduces nonlinear transformations using activation functions such as ReLU and Sigmoid, enabling automatic learning of hierarchical feature representations of input data. MLP enhances model capacity by stacking hidden layers, and the final output layer (usually with a Softmax activation function) generates a category probability distribution. It is suitable for structured data classification tasks, but requires feature engineering for unstructured data such as raw images and time series signals. Its advantages lie in its simplicity and versatility.

[0220] The specific steps of S31 are as follows:

[0221] S311: The Mel spectrum graph in the two-dimensional sound sub-dataset is passed through the Mel spectrum graph feature extraction network MSFE-Net to extract the frequency domain features of the audio data and obtain the Mel high-level feature graph F 19 ;

[0222] The present invention designs a Mel-Spectrogram Feature Extraction Network (MSFE-Net), which is divided into five progressive levels; the first layer preliminarily extracts low-level features of the Mel-Spectrogram to capture basic patterns and frequency components; the second layer further extracts more complex low-level features, identifies more detailed frequency patterns and local changes in the Mel-Spectrogram, and displays spectrum changes in the short term; the third layer begins to extract intermediate features, that is, identifies complex structural patterns in the Mel-Spectrogram and begins to focus on higher-dimensional spectrum information. These intermediate features contain more information than low-level features, and the resolution of the obtained feature map is reduced, but the number of channels increases; the fourth layer extracts higher-level features, mainly identifying global patterns, long-term dependencies between frequency and time, and potential change patterns of device states in the Mel-Spectrogram. The resolution of the obtained feature map is further reduced, but the number of channels remains high; the fifth layer extracts the most abstract features, which highly integrate the feature information of all previous levels and are mainly used to capture the overall behavior patterns, long-term trends, and potential anomalies of the device; the overall structure of MSFE-Net is as follows Figure 3 As shown;

[0223] The specific steps of S311 are as follows:

[0224] S3111: Any mel-spectrogram in the two-dimensional sound sub-dataset is represented as F1, which is input to a convolutional layer with 64 3×3 convolution kernels and a stride of 1, and then processed by the ReLU activation function and batch normalization (i.e. Figure 3 The BN operation in (BN stands for BatchNormalization) is performed to obtain the Mel shallow feature map F2; then, F2 is input into a convolutional layer with 64 3×3 convolution kernels and a stride of 1, and then sequentially passes through the ReLU activation function and batch normalization processing to obtain the Mel shallow feature map F3;

[0225] S3112: Input F3 into the SAPM module to obtain the Mel shallow feature map F4;

[0226] The present invention designs a spatial adaptive pooling module SAPM (SpatialAdaptive Pooling module), wherein the overall structure of SAPM is as follows Figure 4 As shown;

[0227] The Mel shallow feature map F3 is used as the input feature map of the SAPM module; for the convenience of description, F3 is denoted as A1 here;

[0228] Input Mel shallow feature map A1 of size H×W×C, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map; after A1 passes through the SAPM module, a Mel shallow feature map A2 of size H / 2×W / 2×C is obtained. The specific operation is: first, split A1 according to the channel dimension (i.e. Figure 4 The channel dimension splitting operation ChanSplit in , obtains C independent single channel feature maps a 1_1 ,…,A 1_p ,…,A 1_z ; Then each single channel feature map a 1_p (p∈[1,z]), respectively, through the single-channel spatial adaptive pooling module SSAPM (Single-channel Spatial Adaptive Poolingmodule), to obtain the single-channel feature map A with the same channel number 2_p ; Finally, the pooled C single-channel feature maps A 2_1 ,…,A 2_p ,…,A 2_z Reorganize according to channel dimension (i.e. Figure 4 Channel dimension reorganization operation ChanMerge in , to obtain A2;

[0229] The following takes the pth branch as an example to illustrate the conversion of the single-channel feature map a 1_p After the SSAPM module, a single-channel feature map A of size H / 2×W / 2 is obtained 2_p The operation process of the remaining branches is the same as that of the p-th branch; the overall structure of SSAPM is as follows Figure 5 As shown; the input size is H×W single channel feature map A 1_p into the seven branches of the SSAPM module;

[0230] The specific steps of S3112 are as follows:

[0231] S31121: In the first branch, calculate A along the vertical axis 1_p The mean (i.e. Figure 5 The vertical mean vector operation Average_H) in the calculation is used to obtain the vertical mean vector μ p,h ; In the second branch, calculate A along the vertical axis 1_p The variance (i.e. Figure 5 The vertical variance vector operation Variance_H) is obtained by

[0232] μp,h =[μ p,h [1],μ p,h [2],…,μ p,h [h]]

[0233]

[0234] In the above formula, A 1_p (i,j) represents the 1_p The eigenvalue of the (i, j)th position in ;

[0235] S31122: In the third branch, according to A 1_p The output height H is calculated from the height H 2_p (Right now Figure 5 Output height operation Calculate_H);

[0236]

[0237] S31123: In the fourth branch, A 1_p Do any operation; in the fifth branch, calculate A along the horizontal axis 1_p The mean (i.e. Figure 5 The water average vector operation Average_W) in the calculation is used to obtain the water average vector μ p,w , in the sixth branch, calculate A along the horizontal axis 1_p The variance (i.e. Figure 5 The horizontal variance vector operation Variance_W) is obtained by

[0238] μ p,w =[μ p,w [1],μ p,w [2],…,μ p,w [w]]

[0239]

[0240]

[0241] In the above formula, A 1_p (i,j) represents the 1_p The eigenvalue of the (i, j)th position in ;

[0242] S31124: In the seventh branch, according to A 1_p The width W is calculated to get the output width W 2_p (Right now Figure 5 Output width operation Calculate_W in the function);

[0243]

[0244] S31125: Based on variance and Calculate and get the variance factor Based on μ p,h Perform variance guided fusion calculation to obtain the mean μ h , then according to Will Perform variance guided fusion calculation to obtain variance (Right now Figure 5 Variance guided fusion operation VarFusion in μ p,w Perform variance guided fusion calculation to obtain the mean μ w , then according to Will Perform variance guided fusion calculation to obtain the mean (Right now Figure 5 Variance guided fusion operation VarFusion in ); then, according to μ h and Calculate and get the vertical pooling window size k h ; According to μ w and Calculate and get the horizontal pooling window size k w ;

[0245]

[0246] In the above formula, max() represents the operation of calculating the maximum value; mean() represents the operation of calculating the average value; round() represents the operation of rounding the number to the nearest integer; k min is the size of the minimum pooling window, which is used to avoid the pooling window being too small; ε is a constant used to prevent instability caused by the mean being close to zero or the variance being too small;

[0247] S31126: According to k h and H 2_p Calculate the vertical step length s h ; Then, according to k w and W 2_p Calculate the horizontal step size s w ;

[0248]

[0249] In the above formula, round() means rounding the number to the nearest integer;

[0250] S31127: According to A1_p Height H, output height H 2_p , vertical pooling window size k h and vertical step length s h , calculate the vertical padding h According to A 1_p Width W, output width W 2_p , horizontal pooling window size k w and horizontal step length s w , calculate the horizontal padding w ;

[0251]

[0252] Next, for A 1_p Perform zero padding on all four sides in sequence: first in A 1_p Add padding to the left w Zero fill the column and then add padding on the right w Zero fill the columns, then add padding on top h Zero padding the rows, then adding padding at the bottom h The row is zero-filled; the final result is a size of (H+2*padding h )×(W+2*padding w )’s four-sided zero-filled feature map;

[0253] Finally, according to the calculated pooling window size k h 、k w and step length s h 、s w , perform the maximum pooling operation on the feature map with zero padding on all four sides (i.e. Figure 5 The maximum pooling operation MaxPool in the , gets a single channel feature map A of size H / 2×W / 2 2_p ;

[0254] Example:

[0255] Input Mel shallow feature map A1 of size 224×224×64 (where the height H of the feature map is 224, the width W of the feature map is 224, and the number of channels C of the feature map is 64); after A1 passes through the SAPM module, a Mel shallow feature map A2 of size 112×112×64 is obtained. The specific operation is as follows: First, A1 is split according to the channel dimension to obtain 64 independent single-channel feature maps a 1_1 ,…,A 1_p ,…,A 1_64 ; Then each single channel feature map A 1_p(p∈[1,64]) are respectively passed through the single-channel spatial adaptive pooling module SSAPM to obtain the single-channel feature map A with the same channel number 2_p (p∈[1,64]); Finally, the 64 single-channel feature maps A after pooling 2_1 ,…,A 2_p ,…,A 2_64 Reorganize according to the channel dimension to obtain A2;

[0256] Taking the pth (p∈[1,64]) branch as an example, it is shown that the single-channel feature map A 1_p After the SSAPM module, the single-channel feature map A with the same channel number is obtained 2_p The operation process:

[0257] Input single-channel feature map A with size 224×224 1_p into the seven branches of the SSAPM module;

[0258] In the first branch, A is calculated along the vertical axis 1_p The mean of the vertical mean vector μ p,h ;

[0259] μ p,h =[0.11,…,0.09,…,0.3]

[0260] In the second branch, a is calculated along the vertical axis 1_p The variance of the vertical variance vector

[0261]

[0262] In the third branch, according to A 1_p The output height H is calculated from the height H 2_p ;

[0263]

[0264] In the fourth branch, A 1_p Perform any operation;

[0265] In the fifth branch, A is calculated along the horizontal axis. 1_p The mean of the level vector μ p,w ;

[0266] μ p,w =[0.09,…,0.3,…,0.14

[0267] In the sixth branch, A is calculated along the horizontal axis. 1_p The variance of the horizontal variance vector

[0268]

[0269] In the seventh branch, according to A 1_p The width W is calculated to get the output width W 2_p ;

[0270]

[0271] According to the variance and Calculate and get the variance factor

[0272]

[0273] according to μ p,h Perform variance guided fusion calculation to obtain the mean μ h ;

[0274]

[0275] according to Will Perform variance guided fusion calculation to obtain variance

[0276]

[0277] according to μ p,w Perform variance guided fusion calculation to obtain the mean μ w ;

[0278]

[0279] according to Will Perform variance guided fusion calculation to obtain the mean

[0280]

[0281] According to μ h and Calculate and get the vertical pooling window size k h ;

[0282]

[0283] According to μ w and Calculate and get the horizontal pooling window size k w ;

[0284]

[0285] In the above formula, k min Take 2; take ε as 10 -6 ;

[0286] According to k h and H 2_p Calculate the vertical step length s h ;

[0287]

[0288] According to k w and W 2_p Calculate the horizontal step size s w ;

[0289]

[0290] According to A 1_p Height H, output height H 2_p , vertical pooling window size k h and vertical step length s h , calculate the vertical padding h ;

[0291]

[0292] According to A 1_p Width W, output width W 2_p , horizontal pooling window size k w and horizontal step length s w , calculate the horizontal padding w ;

[0293]

[0294] Next, for A 1_p Perform zero padding on all four sides in sequence: first in A 1_p Add 1 column of zero padding on the left, then add 1 column of zero padding on the right, then add 2 rows of zero padding on the top, and then add 2 rows of zero padding on the bottom; finally, the feature map with a size of 228×226 is obtained with zero padding on all four sides;

[0295] Finally, according to the calculated vertical pooling window size k h =5, horizontal pooling window size k w =4 and vertical step size s h =2, horizontal step length s w = 2, perform the maximum pooling operation on the feature map with zero padding on all four sides to obtain a single-channel feature map A of size 112×112 2_p ;

[0296] S3113: Input F4 to a convolutional layer with 128 3×3 convolution kernels and a stride of 1, and then pass through the ReLU activation function and batch normalization to obtain the Mel shallow feature map F5; then input F5 to a convolutional layer with 128 3×3 convolution kernels and a stride of 1, and then pass through the ReLU activation function and batch normalization to obtain the Mel shallow feature map F6; then, input F6 to the SAPM module to obtain the Mel shallow feature map F7;

[0297] S3114: Input F7 to a convolutional layer with 256 3×3 convolution kernels and a stride of 1, and then pass through the ReLU activation function and batch normalization to obtain the Mel middle layer feature map F8; then input F8 to a convolutional layer with 256 3×3 convolution kernels and a stride of 1, and then pass through the ReLU activation function and batch normalization to obtain the Mel middle layer feature map F9; then, input F9 to the SAPM module to obtain the Mel middle layer feature map F 10 ;

[0298] S3115: F 10 The input is sent to a convolutional layer with 512 3×3 convolution kernels and a stride of 1, and then processed by ReLU activation function and batch normalization to obtain the Mel high-level feature map F 11 ; Then F 11 The input is sent to a convolutional layer with 512 3×3 convolution kernels and a stride of 1, and then processed by ReLU activation function and batch normalization to obtain the Mel high-level feature map F 12 ;

[0299] S3116: F 12 Input to the LGFA module to obtain the Mel high-level feature map F 13 ;

[0300] The present invention designs a local-global fusion attention module LGFA (Local-global FusionAttention module), where the overall structure of LGFA is as follows Figure 6 As shown;

[0301] Mel high-level feature map F 13 As the input feature map of the LGFA module. For the convenience of description, F 13 Denoted as B1;

[0302] Input the Mel high-level feature map B1 of size H×W×C to the five branches, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map;

[0303] The specific steps of S3116 are as follows:

[0304] S31161: In the first branch, no operation is performed on B1; in the second branch, global average pooling is first performed on B1 (i.e. Figure 6 GAP operation in ), we get the local attention feature map B2; using a fully connected layer (i.e. Figure 6 FC1 module in ) performs dimensionality reduction operation on B2 to obtain local attention feature map B3; then B3 is activated by ReLU function to obtain local attention feature map B4; then another fully connected layer (i.e. Figure 6 FC2 module in the FC2 module) performs a dimension-up operation on B4 to obtain a local attention feature map B5, so that the number of channels of B5 is consistent with that of B2; then B5 is activated by the Sigmoid function to obtain a local attention feature map B6; then B6 is element-wise multiplied with B1 (i.e. Figure 6 ) to obtain the local attention feature map B7; then, B7 is globally average pooled (i.e. Figure 6 The GAP operation in the pooling is then performed using a multi-layer perceptron to transform the pooling results (i.e. Figure 6 The MLP module in the , the feature transformation result is passed through the Sigmoid activation function to obtain the local attention feature map B8; finally, B7 and B8 are element-wise multiplied (i.e. Figure 6 Element-wise multiplication operation ⊙) in , we get the local attention feature map B9;

[0305] B7=B1⊙S(FC2(ReLU(FC1(GAP(B1)))))

[0306] B9=B7⊙S(MLP(GAP(B7)))

[0307] In the above formula, S represents the Sigmoid activation function; GAP() represents the GlobalAvg Pool operation; ReLU represents the ReLU activation function; MLP() represents the multi-layer perceptron processing; FC1() represents the dimensionality reduction processing through the fully connected layer; FC2() represents the dimensionality increase processing through the fully connected layer; ⊙ represents the element-by-element multiplication operation;

[0308] Global Average Pooling (GAP) is a commonly used feature dimensionality reduction operation. Its core idea is to calculate the average value of all spatial positions (height × width) of each channel of the feature map, thereby compressing a two-dimensional feature map of any size into a one-dimensional vector. Specifically, for a feature map with an input size of H × W × C, GAP calculates the average value of H × W pixels channel by channel, and finally outputs a one-dimensional vector of length C, where each element corresponds to the global response strength of a channel.

[0309] S31162: In the third branch, the weight matrix W v Multiply by B1 (i.e. Figure 6 The matrix multiplication operation × in the , obtains the value matrix V; in the fourth branch, the weight matrix W q Multiply by B1 (i.e. Figure 6 The matrix multiplication operation × in the query matrix Q is obtained; in the fifth branch, the weight matrix W is k Multiply by B1 (i.e. Figure 6 The matrix multiplication operation × in the matrix is used to obtain the key matrix K; then, the dot product of Q and K is calculated and activated by the Softmax function (i.e. Figure 6 The attention score matrix operation QKS in the calculation is obtained to obtain the attention score matrix S; then S is multiplied by V (i.e. Figure 6 The matrix multiplication operation × in the , we get the global attention feature map B 10 ; Then use the convolution kernel of size 1×1 to B 10 Perform convolution operation (i.e. Figure 6 The convolution operation Conv1×1 in the conv1×1 function is used to obtain the global attention feature map B. 11 ; Then, for B 11 Perform global average pooling (i.e. Figure 6 The GAP operation in the pooling is then performed using a multi-layer perceptron to transform the pooling results (i.e. Figure 6 The MLP module in the , the feature transformation result is passed through the Sigmoid activation function to obtain the global attention feature map B 12 ; Then, B 11 With B 12 Perform element-wise multiplication (i.e. Figure 6 , we can get the global attention feature map B by performing the element-wise multiplication operation ⊙ in 13 ;

[0310] V=W v ×B1

[0311] Q=W q ×B1

[0312] K=W k ×B1

[0313]

[0314] B 12 =S(MLP(GAP(B 11 )))

[0315] B 13 =B 11 ⊙B 12

[0316] In the above formula, S represents the Sigmoid activation function; Softmax represents the Softmax activation function; GAP() represents the Global Avg Pool operation; MLP() represents the multi-layer perceptron processing; the He initialization method is used to initialize the weight matrix W. q 、W k and W v Initialize, W q 、W k and W v The dimensions of Where C' is the number of channels after compression; × represents matrix multiplication; ⊙ represents element-by-element multiplication operation; + represents element-by-element addition operation;

[0317] S31163: Finally, B9 and B 13 Perform element-wise addition (i.e. Figure 6 +), and output the fused attention feature map B. 14 ;

[0318] Among them: He initialization is a weight initialization method. Its core idea is to ensure that the input and output of each layer maintain similar variance, thereby avoiding the problem of gradient disappearance or explosion; the steps to generate weights according to the He initialization formula are:

[0319] The first step is to calculate the standard deviation:

[0320]

[0321] The second step is to generate the weight matrix: each element is randomly sampled from the normal distribution N(0,σ) to obtain W q 、W k and W v ;

[0322] In addition, the subsequent change process of the weight is: the weight matrix W q 、W k and W v The value of is updated through back propagation and optimization algorithms (such as SGD, Adam), and its update follows the rules of the optimization algorithm:

[0323]

[0324] In the above formula, η is the learning rate, and is the gradient of the loss function with respect to the weight matrix;

[0325] Example:

[0326] Input the Mel high-level feature map B1 of size 28×28×512 to five branches. In the first branch, no operation is performed on B1. In the second branch, B1 is first globally average pooled to obtain the local attention feature map B2, the size of which is 1×1×512. A fully connected layer is used to reduce the dimension of B2 to obtain the local attention feature map B3, the size of which is 1×1×128. B3 is then activated by the ReLU function to obtain the local attention feature map B4, the size of which is 1×1×128. Another fully connected layer is then used to increase the dimension of B4 to obtain the local attention feature map B5, the size of which is 1×1 ×512; then B5 is activated by the Sigmoid function to obtain the local attention feature map B6, the size of B6 is 1×1×512; then B6 and B1 are multiplied element by element to obtain the local attention feature map B7, the size of B7 is 28×28×512; then B7 is globally averaged pooled; the pooling result is transformed by the multi-layer perceptron, and the feature transformation result is activated by the Sigmoid function to obtain the local attention feature map B8, the size of B8 is 1×1×512; finally, B7 and B8 are multiplied element by element to obtain the local attention feature map B9, the size of B9 is 28×28×512;

[0327] In the third branch, the weight matrix W v Multiply it with B1 to get the value matrix V, the size of V is 28×28×128; in the fourth branch, the weight matrix W q Multiply it with B1 to get the query matrix Q, the size of Q is 28×28×128; in the fifth branch, the weight matrix W k Multiply it with B1 to get the key matrix K, the size of K is 28×28×128; then calculate the dot product of Q and K and pass it through the Softmax activation function to get the attention score matrix S, the size of S is 28×28×28; then multiply S with V to get the global attention feature map B 10 , B 10 The size of B is 28×28×128; then a convolution kernel of size 1×1 is applied to B 10 Perform convolution operation to obtain the global attention feature map B 11 , B 11 The size of B is 28×28×512; then, 11 Perform global average pooling, then use a multi-layer perceptron to transform the pooling result, and then pass the feature transformation result through the Sigmoid activation function to obtain the global attention feature map B 12 , B 12 The size of B is 1×1×512; then, 11 With B 12Perform element-by-element multiplication to obtain the global attention feature map B 13 , B 13 The size is 28×28×512;

[0328] Finally, B9 and B 13 Perform element-by-element addition to obtain and output the fused attention feature map B 14 , B 14 The size is 28×28×512;

[0329] S3117: F 13 Input to the SAPM module to obtain the Mel high-level feature map F 14 ; Then F 14 The input is sent to a convolutional layer with 512 3×3 convolution kernels and a stride of 1, and then processed by ReLU activation function and batch normalization to obtain the Mel high-level feature map F 15 ; Then, F 15 The input is sent to a convolutional layer with 512 3×3 convolution kernels and a stride of 1, and then processed by ReLU activation function and batch normalization to obtain the Mel high-level feature map F 16 ; Then, F 16 Input to the LGFA module to obtain the Mel high-level feature map F 17 ; Next, F 17 Input to the SAPM module to obtain the Mel high-level feature map F 18 ; Finally, F 18 Input to the fully connected layer, obtain and output Mel high-level feature map F 19 ;

[0330] S31 embodiment:

[0331] The Mel-spectrogram F1 in the two-dimensional sound sub-dataset obtained by S2 is used as the input of the Mel-spectrogram feature extraction network. The size of F1 is 224×224 pixels and the number of channels is 1. F1 is input to a convolutional layer with 64 3×3 convolution kernels and a stride of 1, and then passes through the ReLU activation function and the BN layer in sequence to obtain the Mel-shallow feature map F2. The size of F2 is 224×224 pixels and the number of channels is 64. Then, F2 is input to a convolutional layer with 64 3×3 convolution kernels and a stride of 1, and then passes through the ReLU activation function and the BN layer in sequence to obtain the Mel-shallow feature map F3. The size of F3 is 224×224 pixels and the number of channels is 64. F3 is input to the SAPM module to obtain the Mel-shallow feature map F4. The size of F4 is 112×112 pixels and the number of channels is 64.

[0332] F4 is input into a convolutional layer with 128 3×3 convolution kernels and a stride of 1, and then passes through the ReLU activation function and the BN layer in sequence to obtain the Mel shallow feature map F5, the size of F5 is 112×112 pixels, and the number of channels is 128; then F5 is input into a convolutional layer with 128 3×3 convolution kernels and a stride of 1, and then passes through the ReLU activation function and the BN layer in sequence to obtain the Mel shallow feature map F6, the size of F6 is 112×112 pixels, and the number of channels is 128; then, F6 is input into the SAPM module to obtain the Mel shallow feature map F7, the size of F7 is 56×56 pixels, and the number of channels is 128;

[0333] F7 is input into a convolutional layer with 256 3×3 convolution kernels and a stride of 1, and then passes through the ReLU activation function and the BN layer in sequence to obtain the Mel middle layer feature map F8, the size of F8 is 56×56 pixels, and the number of channels is 256; then F8 is input into a convolutional layer with 256 3×3 convolution kernels and a stride of 1, and then passes through the ReLU activation function and the BN layer in sequence to obtain the Mel middle layer feature map F9, the size of F9 is 56×56 pixels, and the number of channels is 256; then, F9 is input into the SAPM module to obtain the Mel middle layer feature map F 10 , F 10 The size is 28×28 pixels and the number of channels is 256;

[0334] F 10 The input is sent to a convolutional layer with 512 3×3 convolution kernels and a stride of 1, and then passes through the ReLU activation function and the BN layer in sequence to obtain the Mel high-level feature map F 11 , F 11 The size of F is 28×28 pixels and the number of channels is 512; then 11 The input is sent to a convolutional layer with 512 3×3 convolution kernels and a stride of 1, and then passes through the ReLU activation function and the BN layer in sequence to obtain the Mel high-level feature map F 12 , F 12 The size of F is 28×28 pixels and the number of channels is 512; then, 12 Input to the LGFA module to obtain the Mel high-level feature map F 13 , F 13 The size of F is 28×28 pixels and the number of channels is 512; 13 Input to the SAPM module to obtain the Mel high-level feature map F 14 , F 14 The size is 14×14 pixels and the number of channels is 512;

[0335] F 14The input is sent to a convolutional layer with 512 3×3 convolution kernels and a stride of 1, and then passes through the ReLU activation function and the BN layer in sequence to obtain the Mel high-level feature map F 15 , F 15 The size of F is 14×14 pixels and the number of channels is 512; then 15 The input is sent to a convolutional layer with 512 3×3 convolution kernels and a stride of 1, and then passes through the ReLU activation function and the BN layer in sequence to obtain the Mel high-level feature map F 16 , F 16 The size of F is 14×14 pixels and the number of channels is 512; then, 16 Input to the LGFA module to obtain the Mel high-level feature map F 17 , F 17 The size of F is 14×14 pixels and the number of channels is 512. 17 Input to the SAPM module to obtain the Mel high-level feature map F 18 , F 18 The size of F is 7×7 pixels and the number of channels is 512. 18 Input to the fully connected layer, obtain and output Mel high-level feature map F 19 , F 19 The feature dimension is 512;

[0336] S312: Extracting time domain features of audio data: The normalized audio data in the one-dimensional sound sub-dataset is input into the bidirectional long short-term memory network BiLSTM; first, the forward LSTM layer in the BiLSTM processes the audio data from left to right according to the time step, gradually updating the forward hidden state, thereby capturing the forward dependency of the audio data in the time series; at the same time, the reverse LSTM layer in the BiLSTM processes the audio data from right to left according to the time step, updating the reverse hidden state, thereby capturing the backward dependency of the audio data; next, the forward and reverse hidden states are concatenated at each time step to obtain the bidirectional features of the time step; finally, the bidirectional features of all time steps are globally averaged and pooled to obtain the time domain feature map E1;

[0337] The Bidirectional Long Short-Term Memory (BiLSTM) network is a recurrent neural network structure that combines forward and backward LSTMs, capable of simultaneously capturing forward and backward dependencies in time series data. Its core consists of two independent LSTM layers: the forward LSTM layer processes the input sequence in chronological order, learning the influence of historical information on the current state; the backward LSTM layer processes the sequence in reverse order, modeling the dependency of future information on the current state. The forward and backward hidden states of each time step are fused through concatenation or summation to form a feature representation that contains complete contextual information. Compared to unidirectional LSTM, BiLSTM significantly improves sequence modeling capabilities through its bidirectional information flow, making it particularly suitable for tasks requiring global context awareness.

[0338] S313: F 19 The audio feature map of the power equipment audio data is obtained by splicing E1 and E2 in the feature dimension. Then, the audio feature map of the power equipment audio data is classified by the MLP classifier to obtain a two-dimensional vector containing the audio fault logits score and the audio fault-free logits score. Finally, the two-dimensional vector is normalized by the Softmax activation function to obtain the audio prediction fault probability. and audio prediction failure-free probability

[0339] S32, anomaly detection based on vibration signals of power equipment: First, the Gram angle field map in the two-dimensional vibration sub-dataset of S2 is used to extract the spatial features of the vibration signal through GAFE-Net; and the normalized vibration signal in the one-dimensional vibration sub-dataset of S2 is used to extract the time series features of the vibration signal through BiGRU. Subsequently, the two results are combined in the feature layer to perform a time series joint feature fusion to obtain the vibration feature map of the power equipment vibration signal. Then, the vibration feature map of the power equipment vibration signal is classified using the MLP classifier. Finally, the classification results are converted into the vibration prediction fault probability and the vibration prediction non-fault probability using the Softmax activation function.

[0340] The specific steps of S32 are as follows:

[0341] S321: The Gram angular field map in the two-dimensional vibration sub-dataset is passed through the Gram angular field map feature extraction network GAFE-Net to extract the spatial features of the vibration signal and obtain the Gram extraction feature map G 15 ;

[0342] The present invention designs a Gramian Angular Field Feature Extraction Network (GAFE-Net), where the overall structure of GAFE-Net is as follows: Figure 7 As shown;

[0343] The specific steps of S321 are as follows:

[0344] S3211: Denote any Gram angular field map in the two-dimensional vibrator dataset as G1, input it into a convolution layer with a convolution kernel size of 4×4 and a stride of 4, and then perform layer normalization to obtain a Gram extraction feature map G2; then, pass G2 through three Bottleneck modules in succession to obtain a Gram extraction feature map G3; then, pass G3 through a Downsample module to reduce its height and width by half while keeping the number of channels unchanged, to obtain a Gram extraction feature map G4. The specific operation is as follows: first, pass G3 through layer normalization, and then pass the layer normalization result through a convolution layer with a convolution kernel size of 2×2 and a stride of 2 to obtain G4; then, pass G4 through five Bottleneck modules in succession to obtain a Gram extraction feature map G5;

[0345] Among them: LN (Layer Normalization) is a layer normalization technique used to stabilize and accelerate the training process of neural networks. Unlike Batch Normalization, Layer Normalization does not rely on the statistical information of small batches of data, but instead performs normalization operations on the features of a single sample. Specifically, for each sample, Layer Normalization calculates the mean and standard deviation of all features of the sample, and then normalizes each feature to have zero mean and unit variance. This normalization method can effectively reduce internal covariate shift and improve the training speed and stability of the model.

[0346] The Bottleneck module is an efficient convolutional neural network structural unit. It consists of three convolutional layers: the first is a 1×1 convolutional layer for dimensionality reduction, which compresses the number of channels in the input feature map and reduces the amount of subsequent computation; the second is a 3×3 convolutional layer for feature extraction. Since the number of input channels has been compressed, the amount of computation is also reduced accordingly; the third is a 1×1 convolutional layer for dimensionality increase, which restores the number of channels to an appropriate size to match the subsequent network structure. This structure effectively reduces the computational complexity by reducing the number of channels in the intermediate feature map, while better preserving feature information, making the network more efficient when processing large-scale data.

[0347] S3212: Input G5 into the DAACO module to obtain the Gram extraction feature map G6;

[0348] The present invention designs a dynamic anomaly-aware channel optimization module DAACO (Dynamic Anomaly-Aware Channel Optimization module), wherein the overall structure of DAACO is as follows Figure 8 As shown;

[0349] The Gram extraction feature map G5 is used as the input feature map of the DAACO module. For ease of description, G5 is denoted as L1 here. The Gram extraction feature map L1 is input to the four branches;

[0350] The specific steps of S3212 are as follows:

[0351] S32121: In the first branch, no operation is performed on L1; in the second branch, global average pooling is used to perform pooling operation on L1 to obtain the channel optimized feature map L2 (i.e. Figure 8 Global average pooling operation AvgPool in 1×1 ); In the third branch, a pooling window of size 3×3 and a dynamic step size s are first used. D Perform average pooling operation on L1, and then perform global average pooling operation on the average pooling result to obtain the channel optimized feature map L3 (i.e. Figure 8 3×3 two-level pooling operation AvgPool 3×3 ); In the fourth branch, we first use a pooling window of size 5×5 and a dynamic step size s D Perform average pooling operation on L1, and then perform global average pooling operation on the average pooling result to obtain the channel optimized feature map L4 (i.e. Figure 8 5×5 two-level pooling operation AvgPool 5×5 );

[0352]

[0353] L2=GAP(L1)

[0354] L3=GAP(AvgPool3(L1))

[0355] L4=GAP(AvgPool5(L1))

[0356] In the above formula, It is a hyperparameter; max() indicates the maximum value operation; GAP() indicates the Global AvgPool operation; AvgPool3() indicates that the pooling window size is 3×3 and the step size is s D Average pooling operation; AvgPool5() means the pooling window size is 5×5 and the step size is s D Average pooling operation;

[0357] S32122: Set the channel sparse parameter W S1 Multiply by L2 (i.e. Figure 8 The parameter multiplication operation × in the channel is used to obtain the channel optimized feature map L5; the channel sparse parameter WS2 Multiply by L3 (i.e. Figure 8 The parameter multiplication operation × in the channel is used to obtain the channel optimized feature map L7; the channel sparse parameter W S3 Multiply by L4 (i.e. Figure 8 The parameter multiplication operation × in the channel is used to obtain the channel optimized feature map L9; then L5, L7 and L9 are added element by element (i.e. Figure 8 Element-by-element addition operation in +) to obtain the channel optimized feature map L 11 ; Then L 11 After the Sigmoid activation function, the channel optimized feature map L is obtained 12 ; Then L 12 Perform element-wise multiplication with L1 (i.e. Figure 8 Element-wise multiplication operation ⊙) in the channel to obtain the channel optimized feature map L 13 ;

[0358] L 13 =S(W S1 ×L2+W S2 ×L3+W S3 ×L4)⊙L1

[0359] In the above formula, the Xavier initialization method is used to initialize the channel sparse parameter W S1 、W S2 and W S3 Initialize; × represents matrix multiplication; ⊙ represents element-by-element multiplication; + represents element-by-element addition;

[0360] S32123: Set the abnormal channel parameter W A1 Multiply by L2 (i.e. Figure 8 The parameter multiplication operation ×) is used to obtain the channel optimization feature map L6; the abnormal channel parameter W A2 Multiply by L3 (i.e. Figure 8 The parameter multiplication operation ×) is used to obtain the channel optimization feature map L8; the abnormal channel parameter W A3 Multiply by L4 to get the channel optimized feature map L 10 ; Then L6, L8 and L 10 Perform element-wise addition (i.e. Figure 8 The parameter multiplication operation ×) in the channel optimization feature map L is obtained 14 ; Then L 14 After the ReLU activation function, the channel optimized feature map L is obtained 15 ;

[0361] L 15 =ReLU(W A1 ×L6+W A2 ×L8+W A3 ×L10 )

[0362] In the above formula, Gaussian distribution is used Initialization method, abnormal channel parameter W A1 、W A2 and W A3 Initialize; ReLU represents the ReLU activation function; × represents matrix multiplication; + represents element-by-element addition operation;

[0363] S32124: L 13 With L 15 Perform element-wise multiplication (i.e. Figure 8 Element-wise multiplication operation ⊙) in the channel to obtain the channel optimized feature map L 16 ; Afterwards, the channel attention parameter W D With L 16 Perform multiplication (i.e. Figure 8 The parameter multiplication operation ×) in the channel optimization feature map L is obtained 17 ; Then L 17 After the Softmax activation function, the channel optimized feature map L is obtained 18 ; Finally, L 16 With L 18 Perform element-wise multiplication (i.e. Figure 8 Element-by-element multiplication operation ⊙) in the channel optimization feature map L is obtained and output. 19 ;

[0364] L 19 =L 16 ⊙Softmax(W D ×(L 13 ⊙L 15 ))

[0365] In the above formula, Gaussian distribution is used Initialization method, channel attention parameter W D Initialize; Softmax represents the Softmax activation function; × represents matrix multiplication; ⊙ represents element-by-element multiplication operation;

[0366] Where: (1) Xavier initialization

[0367] Xavier initialization keeps the initial weight value small to prevent gradient disappearance or gradient explosion. It is suitable for networks with Sigmoid or Tanh activation functions. The formula is:

[0368]

[0369] In the above formula, n in is the number of input channels, n out is the number of output channels;

[0370] (2) Gaussian distribution initialization

[0371] Gaussian distribution initialization is suitable for ReLU or LeakyReLU networks to maintain stable gradient updates, where the formula is:

[0372]

[0373] In the above formula, the mean of the Gaussian distribution is 0 and the standard deviation is 0.02, which means that the initial values of the parameters are concentrated around 0, and ~ means that they obey the probability distribution;

[0374] Example:

[0375] The input size is 28×28×128 Gram extraction feature map L1 to four branches; in the first branch, no operation is performed on L1; in the second branch, global average pooling is used to pool L1 to obtain the channel optimized feature map L2, the size of L2 is 1×1×128; in the third branch, a pooling window of size 3×3 and a dynamic step size s are first used. D =2 performs average pooling operation on L1, and then performs global average pooling operation on the average pooling result to obtain the channel optimized feature map L3, the size of L3 is 1×1×128; in the fourth branch, first use a pooling window of size 5×5 and a dynamic step size s D =2 performs average pooling operation on L1, and then performs global average pooling operation on the average pooling result to obtain the channel optimized feature map L4, the size of L4 is 1×1×128; the dynamic step size s D The calculation process is:

[0376]

[0377] Then, the channel sparsity parameter W S1 Multiply by L2 to get the channel optimized feature map L5, the size of L5 is 1×1×128; the channel sparse parameter W S2 Multiply it with L3 to get the channel optimized feature map L7, the size of L7 is 1×1×128; the channel sparse parameter W S3 Multiply it with L4 to get the channel optimized feature map L9, the size of L9 is 1×1×128; then perform element-by-element addition operation on L5, L7 and L9 to get the channel optimized feature map L 11 , L 11 The size of L is 1×1×128; then 11 After the Sigmoid activation function, the channel optimized feature map L is obtained 12 , L 12 The size of L is 1×1×128; then12 Perform element-by-element multiplication with L1 to obtain the channel optimized feature map L 13 , L 13 The size is 28×28×128;

[0378] Secondly, the abnormal channel parameter W A1 Multiply by L2 to get the channel optimization feature map L6, the size of L6 is 1×1×128; the abnormal channel parameter W A2 Multiply it by L3 to get the channel optimized feature map L8, the size of L8 is 1×1×128; the abnormal channel parameter W A3 Multiply by L4 to get the channel optimized feature map L 10 , L 10 The size of L6, L8 and L 10 Perform element-by-element addition to obtain the channel optimized feature map L 14 , L 14 The size of L is 1×1×128; then 14 After the ReLU activation function, the channel optimized feature map L is obtained 15 , L 15 The size is 1×1×128;

[0379] Next, first L 13 With L 15 Perform element-by-element multiplication to obtain the channel optimized feature map L 16 , L 16 The size of is 28×28×128; then, the channel attention parameter W D With L 16 Perform multiplication operation to obtain the channel optimized feature map L 17 , L 17 The size of L is 28×28×128; 17 After the Softmax activation function, the channel optimized feature map L is obtained 18 , L 18 The size of L is 28×28×128; finally, L 16 With L 18 Perform element-by-element multiplication to obtain and output the channel optimized feature map L 19 , L 19 The size is 28×28×128;

[0380] S3213: Input G6 into the Downsample module to reduce its height and width by half while keeping the number of channels unchanged, thereby obtaining the Grammar extraction feature map G7. Then, G7 is passed through seven Bottleneck modules in succession to obtain the Grammar extraction feature map G8. Finally, G8 is passed through the DAACO module to obtain the Grammar extraction feature map G9.

[0381] S3214: Input G9 into the MLAF module to obtain the Gram extraction feature map G 10 ;

[0382] The present invention designs a multi-level adaptive fusion module MLAF (Multi-level Adaptive Fusion module), wherein the overall structure of MLAF is as follows Figure 9 As shown;

[0383] The Gram extraction feature map G9 is used as the input feature map of the MLAF module; for the convenience of description, G9 is denoted as T1 here;

[0384] Input the Gram extraction feature map T1 of size H×W×C to the nine branches, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map;

[0385] The specific steps of S3214 are as follows:

[0386] S32141: In the first branch, no operation is performed on T1; in the second branch, no operation is performed on T1; in the third branch, firstly, wavelet transform is performed on T1 to extract high-frequency abnormal features (i.e. Figure 9 The wavelet transform operation DWT in the image is used to obtain the multi-level fusion feature map T2; then the convolution operation of T2 is performed using a convolution kernel of size 1×1 (i.e. Figure 9 The convolution operation Conv1×1 in the CNN is performed to obtain a multi-level fusion feature map T3; then T3 is activated by the Sigmoid function to obtain a multi-level fusion feature map T4:

[0387] T4 = S(Conv1(DWT(T1)))

[0388] In the above formula, DWT() represents the wavelet transform operation; Conv1() represents the convolution operation of the convolution kernel with a size of 1×1; S represents the Sigmoid activation function;

[0389] The Discrete Wavelet Transform (DWT) is a time-frequency analysis technique that achieves multi-resolution analysis of signals by decomposing them into approximate coefficients (low-frequency information) and detail coefficients (high-frequency information) of different scales. Its core is to convolve and downsample the original signal using orthogonal wavelet basis functions (such as Haar and Daubechies), effectively capturing transient anomalies and local details while preserving the signal's time domain characteristics. Compared to the Fourier transform, the wavelet transform has an adaptive windowing feature, making it suitable for processing non-stationary time series data such as power equipment vibration signals, and can accurately extract fault impulse characteristics and periodic components.

[0390] S32142: In the fourth branch, a depth-wise separable convolution operation is performed on T1 using a convolution kernel of size 1×1 (i.e. Figure 9 The depth-wise separable convolution operation DSC1×1 in the fifth branch is used to obtain the multi-level fusion feature map T5; in the fifth branch, a depth-wise separable convolution operation is performed on T1 using a convolution kernel of size 3×3 (i.e. Figure 9 The depth-wise separable convolution operation DSC3×3 in the CNN is used to obtain the multi-level fusion feature map T6; in the sixth branch, a depth-wise separable convolution operation is performed on T1 using a convolution kernel of size 5×5 (i.e. Figure 9 The depth-wise separable convolution operation DSC5×5 in the CNN is used to obtain the multi-level fusion feature map T7; in the seventh branch, a depth-wise separable convolution operation is performed on T1 using a convolution kernel of size 7×7 (i.e. Figure 9 The depthwise separable convolution operation DSC7×7) in the CNN obtains the multi-level fusion feature map T8;

[0391] Depthwise Separable Convolution is an efficient convolution operation that decomposes the standard convolution into two independent steps to reduce computational complexity. First, a channel-by-channel spatial convolution is performed, using a two-dimensional convolution kernel for each input channel to extract spatial features while keeping the number of channels constant. A 1×1 pointwise convolution is then used to fuse cross-channel information and adjust the output channel dimensions. This structure significantly reduces the number of parameters and computational cost while maintaining good feature extraction capabilities.

[0392] After that, T5, T6, T7 and T8 are added element by element (i.e. Figure 9 The element-by-element addition operation in +) is performed to obtain a multi-level fusion feature map T9; then T9 is activated by the Sigmoid function to obtain a multi-level fusion feature map T 10 :

[0393] T 10 =S(DSC1(T1)+DSC3(T1)+DSC5(T1)+DSC7(T1))

[0394] In the above formula, DSC1() represents the depth-separable convolution operation of the convolution kernel of size 1×1; DSC3() represents the depth-separable convolution operation of the convolution kernel of size 3×3; DSC5() represents the depth-separable convolution operation of the convolution kernel of size 5×5; DSC7() represents the depth-separable convolution operation of the convolution kernel of size 7×7; + represents the element-by-element addition operation; S represents the Sigmoid activation function;

[0395] S32143: In the eighth branch, perform an average pooling operation on T1 in the channel dimension (i.e. Figure 9 The average pooling operation on the channel dimension in Channel Avg Pool is as follows: traverse each spatial position (i, j) of T1 (where i∈[1,H], j∈[1,W]), and calculate the average value of its position on all channels; in the ninth branch, perform the maximum pooling operation on T1 in the channel dimension (i.e. Figure 9 The maximum pooling operation on the channel dimension in Channel Max Pool is as follows: traverse each spatial position (i, j) of T1 (where i∈[1,H], j∈[1,W]), calculate the maximum value of its position on all channels; then splice the results of the eighth branch and the ninth branch on the channel dimension (i.e. Figure 9 Perform concatenation operation Concat in the channel dimension to obtain a multi-level fusion feature map T 11 (Right now Figure 9 The convolution operation Conv1×1 in T is then applied with a convolution kernel of size 1×1. 11 Perform convolution operation to obtain a multi-level fusion feature map T 12 ; Then use the convolution kernel of size 7×7 to T 12 Perform convolution operation (i.e. Figure 9 The convolution operation Conv7×7 in the convnet is used to obtain the multi-level fusion feature map T 13 ; Use a convolution kernel of size 3×3 and a dilation rate of 2 to T 12 Perform convolution operation (i.e. Figure 9 The convolution operation Conv3×3, Rate=2) in the CNN obtains the multi-level fusion feature map T 14 ; Then T 13 and T 14 Perform element-wise addition (i.e. Figure 9 Element-by-element addition operation in +) to obtain the multi-level fusion feature map T 15 ; Then T 15 After the Sigmoid activation function, the multi-level fusion feature map T is obtained 16 :

[0396]

[0397] T 15 =S(Conv7(T 12 )+Conv3_2(T 12 ))

[0398] In the above formula, T1(i,j,k) represents the eigenvalue of T1 at the spatial position (i,j) and the kth channel; max k∈[1,C]() indicates the operation of calculating the maximum value of each spatial position in all channels; Concat() indicates the concatenation operation in the channel dimension; Conv7() indicates the convolution operation of a convolution kernel of size 7×7; Conv3_2() indicates the convolution operation of a convolution kernel of size 3×3 and a void rate of 2; S indicates the Sigmoid activation function; + indicates the element-by-element addition operation;

[0399] S32144: T1, T4, T 10 and T 16 Perform element-wise multiplication (i.e. Figure 9 The element-by-element multiplication operation ⊙) in the fusion function is used to obtain the multi-level fusion feature map T 17 ; Then use the convolution kernel of size 1×1 to T 17 Perform convolution operation to obtain a multi-level fusion feature map T 18 ; Then T 18 After the Softmax activation function, the abnormal mask M is obtained a ;

[0400] Afterwards, M a Perform two small branch calculations; in the first small branch, calculate M a The reverse exception mask (i.e. Figure 9 The reverse exception mask operation RevMask in the above example is used to obtain the reverse exception mask M. 1-a ; Then T1 and M 1-a Perform element-wise multiplication (i.e. Figure 9 The element-by-element multiplication operation ⊙) in the fusion function is used to obtain the multi-level fusion feature map T 19 ; In the second branch, T 17 and M a Perform element-wise multiplication (i.e. Figure 9 The element-by-element multiplication operation ⊙) in the fusion function is used to obtain the multi-level fusion feature map T 20 ; Finally, T 19 and T 20 Perform element-wise addition (i.e. Figure 9 +), and obtain and output the multi-level fusion feature map T 21 ;

[0401] M a =Softmax(Conv1(T1⊙T4⊙T 10 ⊙T 16 ))

[0402] M 1-a =1-M a

[0403] T 21 =T1⊙M 1-a+M a ⊙T 17

[0404] In the above formula, Conv1() represents the convolution operation of the convolution kernel with a size of 1×1; Softmax represents the Softmax activation function; ⊙ represents the element-by-element multiplication operation; + represents the element-by-element addition operation;

[0405] Example:

[0406] Input the Gram extraction feature map T1 of size 14×14×256 to nine branches; in the first branch, no operation is performed on T1; in the second branch, no operation is performed on T1; in the third branch, first, wavelet transform is performed on T1 to extract high-frequency abnormal features to obtain a multi-level fusion feature map T2 of size 14×14×256; then a convolution kernel of size 1×1 is used to convolve T2 to obtain a multi-level fusion feature map T3 of size 14×14×256; then T3 is subjected to a Sigmoid activation function to obtain a multi-level fusion feature map T4 of size 14×14×256;

[0407] In the fourth branch, a 1×1 convolution kernel is used to perform a depth-separable convolution operation on T1 to obtain a multi-level fusion feature map T5, the size of which is 14×14×256; in the fifth branch, a 3×3 convolution kernel is used, the step size is set to 1, and the padding is set to 1, and a depth-separable convolution operation is performed on T1 to obtain a multi-level fusion feature map T6, the size of which is 14×14×256; in the sixth branch, a 5×5 convolution kernel is used, the step size is set to 1, and the padding is set to 2, and a depth-separable convolution operation is performed on T1 to obtain To the multi-level fusion feature map T7, the size of T7 is 14×14×256; in the seventh branch, use the convolution kernel of size 7×7, set the step size to 1, and fill to 3, perform a depth-separable convolution operation on T1, and obtain the multi-level fusion feature map T8, the size of T8 is 14×14×256; then, perform element-by-element addition operation on T5, T6, T7 and T8 to obtain the multi-level fusion feature map T9, the size of T9 is 14×14×256; then, T9 is activated by Sigmoid function to obtain the multi-level fusion feature map T 10 , T 10 The size is 14×14×256;

[0408] In the eighth branch, an average pooling operation is performed on T1 in the channel dimension, and the size of the average pooling result is 14×14×1; in the ninth branch, a maximum pooling operation is performed on T1 in the channel dimension, and the size of the maximum pooling result is 14×14×1; then the results of the eighth and ninth branches are spliced in the channel dimension to obtain a multi-level fusion feature map T11 , T 11 The size of T is 14×14×2; then, a convolution kernel of size 1×1 is applied to T 11 Perform convolution operation to obtain a multi-level fusion feature map T 12 , T 12 The size of the convolution kernel is 14×14×256; then the convolution kernel of size 7×7 is used, the step size is set to 1, the padding is set to 3, and the T 12 Perform convolution operation to obtain a multi-level fusion feature map T 13 , T 13 The size of the convolution kernel is 14×14×256; the convolution kernel with a size of 3×3 and a void rate of 2 is used, the step size is set to 1 and the padding is set to 2. 12 Perform convolution operation to obtain a multi-level fusion feature map T 14 , T 14 The size of T is 14×14×256; then 13 and T 14 Perform element-by-element addition to obtain a multi-level fusion feature map T 15 , T 15 The size of T is 14×14×256; 15 After the Sigmoid activation function, the multi-level fusion feature map T is obtained 16 , T 16 The size is 14×14×256;

[0409] T1, T4, T 10 and T 16 Perform element-by-element multiplication to obtain a multi-level fusion feature map T 17 , T 17 The size of is 14×14×256; then use the convolution kernel of size 1×1 to 17 Perform convolution operation to obtain a multi-level fusion feature map T 18 , T 18 The size is 14×14×1; then T 18 After the Softmax activation function, the abnormal mask M is obtained a , M a The size of M is 14×14×1; a Perform two small branch calculations; in the first small branch, calculate M a The reverse exception mask of 1-a , M 1-a The size is 14×14×1; then T1 and M 1-a Perform element-by-element multiplication to obtain a multi-level fusion feature map T 19 , T 19The size of is 14×14×256; in the second branch, T 17 and M a Perform element-by-element multiplication to obtain a multi-level fusion feature map T 20 , T 20 The size of T is 14×14×256; finally, 19 and T 20 Perform element-by-element addition to obtain and output a multi-level fusion feature map T 21 , T 21 The size is 14×14×256;

[0410] S3215: G 10 Input it to the Downsample module to reduce its height and width by half, while keeping the number of channels unchanged, and obtain the Gram extraction feature map G 11 ; Then, G 11 Continuously pass through 5 Bottleneck modules to obtain the Gram extraction feature map G 12 ; Then, G 12 Through the MLAF module, we get the Gram extraction feature map G 13 ; Then, G 13 Perform global average pooling (i.e. Figure 7 Global AVG Pooling operation in ), get Gram extraction feature map G 14 ; Finally, G 14 Perform layer normalization (i.e. Figure 7 LN operation in), obtain and output Gram extraction feature map G 15 ;

[0411] S32 embodiment:

[0412] The Gram angular field map G1 in the two-dimensional vibrator dataset obtained by S2 is used as the input of the Gram angular field map feature extraction network. The size of G1 is 224×224 pixels and the number of channels is 1. G1 is input into the convolution layer with a convolution kernel size of 4×4 and a stride of 4, and then passes through the LN layer to obtain the Gram extraction feature map G2. The size of G2 is 56×56 pixels and the number of channels is 64. Then, G2 is continuously passed through three Bottleneck modules to obtain the Gram extraction feature map G3. The size of G3 is 56×56 pixels and the number of channels is 64. After that, G3 is passed through the Downsample module to obtain the Gram extraction feature map G4. The size of G4 is 28×28 pixels and the number of channels is 128.

[0413] Then, G4 is passed through five Bottleneck modules to obtain the Grammar extraction feature map G5, which has a size of 28×28 pixels and a number of channels of 128. G5 is input into the DAACO module to obtain the Grammar extraction feature map G6, which has a size of 28×28 pixels and a number of channels of 128. G6 is input into the Downsample module to obtain the Grammar extraction feature map G7, which has a size of 14×14 pixels and a number of channels of 256.

[0414] Then, G7 is passed through 7 Bottleneck modules in succession to obtain the Gram extraction feature map G8, the size of G8 is 14×14 pixels, and the number of channels is 256; then, G8 is passed through the DAACO module to obtain the Gram extraction feature map G9, the size of G9 is 14×14 pixels, and the number of channels is 256; G9 is input into the MLAF module to obtain the Gram extraction feature map G 10 , G 10 The size of G is 14×14 pixels and the number of channels is 256; 10 Input to the Downsample module to obtain the Gram extraction feature map G 11 , G 11 The size is 7×7 pixels and the number of channels is 512;

[0415] Then, G 11 Continuously pass through 5 Bottleneck modules to obtain the Gram extraction feature map G 12 , G 12 The size of G is 7×7 pixels and the number of channels is 512; then, 12 Through the MLAF module, we get the Gram extraction feature map G 13 , G 13 The size of G is 7×7 pixels and the number of channels is 512; then, 13 Perform global average pooling to obtain the Gram extraction feature map G 14 , G 14 The size of G is 1×1 pixel and the number of channels is 512; finally, 14 Perform layer normalization to obtain and output the Gram extraction feature map G 15 , G 15 The size is 1×1 pixel and the number of channels is 512;

[0416] S322: Extracting the temporal features of the vibration signal: The normalized vibration signal in the one-dimensional vibration sub-dataset is input into a bidirectional gated recurrent unit (BiGRU). First, the forward GRU layer in the BiGRU processes the vibration signal from left to right in chronological order, and gradually updates the forward hidden state using a gating mechanism consisting only of an update gate and a reset gate to capture the forward temporal pattern of the vibration signal. Simultaneously, the reverse GRU layer processes the signal inversely from right to left, controlling the flow of information through gating to capture the reverse dependency of the vibration signal. Then, at each time step, the forward and reverse hidden states are concatenated according to the channel dimension to form a fused feature containing bidirectional temporal information. Finally, the fused features of all time steps are globally averaged and pooled to obtain a temporal feature map N1.

[0417] S323: G 15 and N1 in the feature dimension to obtain the vibration feature map of the power equipment vibration signal; then, the vibration feature map of the power equipment vibration signal is classified by the MLP classifier to obtain a two-dimensional vector containing the vibration fault logits score and the vibration fault-free logits score; finally, the two-dimensional vector is normalized by the Softmax activation function to obtain the vibration prediction fault probability and vibration prediction failure-free probability

[0418] S33, anomaly detection based on power equipment images: For power equipment images in image files, ADEI-Net is used to perform target detection on the power equipment images, extract visual features of the equipment, and determine the anomalies of the images. The predicted probability of the images being faulty and the predicted probability of being fault-free are output.

[0419] The present invention designs an electrical equipment image anomaly detection network ADEI-Net (Anomaly Detection of Electrical Equipment Image Network), wherein the overall structure of ADEI-Net is as follows Figure 10 As shown;

[0420] The specific steps of S33 are as follows:

[0421] S331: Denote any power equipment image in the image sub-dataset as P1 and input it into the CBS module to obtain a power equipment detection feature map P2; then, pass P2 through the CBS module to obtain a power equipment detection feature map P3; then, pass P3 through the C2F module to obtain a power equipment detection feature map P4; then, pass P4 through the CBS module to obtain a power equipment detection feature map P5;

[0422] The CBS module consists of three parts connected in series: a 3×3 Conv2d layer with a stride of 2, a BatchNorm2d layer, and a SiLU layer.

[0423] The C2F module consists of two 1×1 Conv2d modules with a stride of 1 and several Bottleneck modules. It introduces skip connections to connect feature maps at different stages, thereby promoting feature transfer and information flow at different levels.

[0424] S332: Input P5 into the CFAE module to obtain the power equipment detection characteristic diagram P6;

[0425] The present invention designs a channel feature adaptive enhancement module CFAE (Channel-Feature Adaptive Enhancement module), wherein the overall structure of CFAE is as follows Figure 11 As shown;

[0426] The power equipment detection feature map P5 is used as the input feature map of the CFAE module. For the sake of convenience, P5 is denoted as V1 here;

[0427] The specific steps of S332 are as follows:

[0428] S3321: Input the power equipment detection feature map V1. First, use the channel shuffling method to perform cross-mixing operations on the channel information of V1 (i.e. Figure 11 Channel Shuffle operation in the channel) to obtain the channel feature enhancement feature map V2; then, the segmented GLU is used to gate V2 (i.e. Figure 11 The GLU part in the channel feature map is obtained by first dividing V2 into two parts V along the channel dimension. 2_1 and V 2_2 (Right now Figure 11 Split operation in the V 2_2 Through the Sigmoid activation function, the gate signal G is obtained, and finally V 2_1 and G perform element-wise multiplication (i.e. Figure 11 , we get V3; then, we use a convolution kernel of size 1×1 to perform a convolution operation on V3 (i.e. Figure 11 The convolution operation Conv1×1 in is used to obtain the channel feature enhanced feature map V4;

[0429] S3322: Perform four branch calculations on V4. In the first branch, global average pooling (i.e. Figure 11The Global Avg Pool operation in the pooling operation is performed on V4 to obtain the channel feature enhancement feature map V5; then a convolution operation is performed on V5 using a convolution kernel of size 1×1 (i.e. Figure 11 The convolution operation Conv1×1 in the convolution operation is performed to obtain the channel feature enhanced feature map V6; then V6 is activated by the Sigmoid function to obtain the channel feature enhanced feature map V7; in the second branch, a convolution operation is performed on V4 using a convolution kernel of size 3×3 (i.e. Figure 11 The convolution operation Conv3×3 in the image is used to obtain the channel feature enhancement feature map V8; V8 is then activated by the Tanh function to obtain the channel feature enhancement feature map V9; in the third branch, no operation is performed on V4; after that, V7, V9 and V4 are element-wise multiplied (i.e. Figure 11 Element-wise multiplication operation ⊙) in the channel feature enhancement feature map V 11 ;

[0430] S3323: In the fourth branch, V4 is extracted through the Bottleneck module to obtain the channel feature enhancement feature map V 10 ; Finally, V 11 and V 10 Perform channel-by-channel maximum fusion (i.e. Figure 11 The channel-by-channel maximum fusion operation MaxFusion is performed to obtain and output the channel feature enhancement feature map V 12 , the specific operation is: for V 11 and V 10 For each corresponding channel, traverse all spatial positions on the channel feature map and calculate V 11 and V 10 The maximum value of the element at that position is used as V 12 Elements in

[0431] Channel shuffling is a technique used to enhance information flow between feature channels. Its core idea is to break the fixed grouping relationship between channels by rearranging the channels of the feature map, thereby promoting information exchange between different channels. Its working principle is as follows: assuming the number of channels of the input feature map is C, it is divided into G groups, each containing C / G channels. First, the channel dimensions of the input feature map are reshaped to (G, C / G, H, W). Then, the group dimensions and channel dimensions are swapped to obtain (C / G, G, H, W). Finally, it is flattened back to the original shape (C, H, W).

[0432] Example:

[0433] The input size is 28×28×256 power equipment detection feature map V1. First, the channel shuffling method is used to perform cross-mixing operations on the channel information of V1 to obtain the channel feature enhancement feature map V2. The size of V2 is 28×28×256. Then, the split GLU is used to perform gate processing on V2 to obtain the channel feature enhancement feature map V3. The size of V3 is 28×28×128. The specific steps are: first, V2 is divided into two parts along the channel dimension V 2_1 and V 2_2 , both sizes are 28×28×128, and then V 2_2 Through the Sigmoid activation function, the gate signal G is obtained, and finally V 2_1 Perform element-by-element multiplication with G; then, use a convolution kernel of size 1×1 to perform a convolution operation on V3 to obtain a channel feature enhanced feature map V4, the size of V4 is 28×28×256;

[0434] Four branches are calculated for V4. In the first branch, global average pooling is first used to perform a pooling operation on V4 to obtain a channel feature enhanced feature map V5, the size of which is 1×1×256. Then, a convolution kernel of size 1×1 is used to perform a convolution operation on V5 to obtain a channel feature enhanced feature map V6, the size of which is 1×1×256. Then, V6 is subjected to a Sigmoid activation function to obtain a channel feature enhanced feature map V7, the size of which is 1×1×256. In the second branch, the convolution kernel of size 1×1 is used to perform a convolution operation on V5 to obtain a channel feature enhanced feature map V6, the size of which is 1×1×256. In the second branch, a convolution operation is performed on V4 using a convolution kernel of size 3×3, with a step size of 1 and a padding of 1, to obtain a channel feature enhancement feature map V8 of size 28×28×256; V8 is then activated by the Tanh function to obtain a channel feature enhancement feature map V9 of size 28×28×256; in the third branch, no operation is performed on V4; after that, V7, V9 and V4 are element-wise multiplied to obtain a channel feature enhancement feature map V 11 , V 11 The size is 28×28×256;

[0435] In the fourth branch, V4 is extracted through the Bottleneck module to obtain the channel feature enhancement feature map V 10 , V 10 The size of V is 28×28×256; finally, 11 and V 10 Perform channel-by-channel maximum fusion to obtain and output the channel feature enhancement feature map V 12 , V 12 The size is 28×28×256;

[0436] S333: Input P6 into the CBS module to obtain the power equipment detection feature map P7; then, pass P7 through the CFAE module to obtain the power equipment detection feature map P8; then, pass P8 through the CBS module to obtain the power equipment detection feature map P9; then, pass P9 through the CFAE module to obtain the power equipment detection feature map P 10 ; Then P 10 Through the SPPF module, the power equipment detection feature map P is obtained 11 ; Then, P 11 Perform upsampling operation (i.e. Figure 10 The up-sampling operation Upsample in the above example is used to obtain the power equipment detection feature map P 12 ; Next, P8 and P 12 Splicing is performed on the channel dimension (i.e. Figure 10 Perform concatenation operation Concat in the channel dimension to obtain the power equipment detection feature map P 13 ; Then P 13 Through the C2F module, the power equipment detection feature map P is obtained 14 ; Then P 14 Perform upsampling operation (i.e. Figure 10 The up-sampling operation Upsample in the above example is used to obtain the power equipment detection feature map P 15 ; Next, P6 and P 15 Splicing is performed on the channel dimension (i.e. Figure 10 Perform concatenation operation Concat in the channel dimension to obtain the power equipment detection feature map P 16 ; Then P 16 Through the C2F module, the power equipment detection feature map P is obtained 17 ; Then P 17 Through the CBS module, the power equipment detection characteristic map P is obtained 18 ; Next, P 14 and P 18 Splicing is performed on the channel dimension (i.e. Figure 10 Perform concatenation operation Concat in the channel dimension to obtain the power equipment detection feature map P 19 ; Then, P 19 Through the C2F module, the power equipment detection feature map P is obtained 20 ; Then, P 20 Through the CBS module, the power equipment detection characteristic map P is obtained 21 ; Then P 11 and P 21 Splicing is performed on the channel dimension (i.e. Figure 10 Perform concatenation operation Concat in the channel dimension to obtain the power equipment detection feature map P 22; Next, P 22 Through the C2F module, the power equipment detection feature map P is obtained 23 ;

[0437] The SPPF module is an efficient multi-scale feature fusion module that quickly captures contextual information of different receptive fields through cascaded maximum pooling operations. The module first performs multiple maximum pooling operations of different sizes on the input feature map, then concatenates the original feature map with the pooling results in the channel dimension, and finally compresses the number of channels through 1×1 convolution. Compared with the traditional SPP module, the SPPF adopts a serial stacking pooling method, which significantly reduces the amount of computation while maintaining the multi-scale feature extraction capability, effectively improving the model's detection performance for targets of different sizes.

[0438] S334: P 17 Input to the AADM module to obtain the power equipment detection feature map P 24 ;P 20 Input to the AADM module to obtain the power equipment detection feature map P 27 ;P 23 Input to the AADM module to obtain the power equipment detection feature map P 30 ;

[0439] This paper designs an attention abnormality detection module AADM (Attention Abnormality Detection module), where the overall structure of AADM is as follows Figure 12 As shown;

[0440] The power equipment detection characteristic map P 17 As the input feature map of the AADM module. For the convenience of description, P 17 Denoted as Y1;

[0441] The specific steps of S334 are as follows:

[0442] S3341: Input the power equipment detection feature map Y1. First, use the convolution kernel of size 3×3 to perform convolution operation on Y1 (i.e. Figure 12 The convolution operation Conv3×3 in the convolution operation is used to perform a convolution operation on Y1 using a convolution kernel of size 5×5 (i.e. Figure 12 The convolution operation Conv5×5 in the convolution operation and the convolution operation of Y1 using the convolution kernel of size 7×7 (ie Figure 12 The convolution operation Conv7×7 in ); after that, the three convolution results are spliced in the channel dimension (i.e. Figure 12 The concatenation operation Concat is performed on the channel dimension to obtain the attention anomaly detection feature map Y2; then a convolution operation of 1×1 convolution kernel is performed on Y2 (i.e. Figure 12 The convolution operation Conv1×1 in

[15] is performed to obtain the attention anomaly detection feature map Y3; then Y3 is first processed by the ReLU activation function, and the processing result is finally processed by the Hard Sigmoid activation function to obtain the attention anomaly detection feature map Y4;

[0443] Y2=Concat(Conv3(Y1),Conv5(Y1),Conv7(Y1))

[0444] Y4=HardS(ReLU(Conv1(Y2)))

[0445] In the above formula, Conv3() represents the convolution operation of the convolution kernel of size 3×3; Conv5() represents the convolution operation of the convolution kernel of size 5×5; Conv7() represents the convolution operation of the convolution kernel of size 7×7; Concat() represents the splicing operation in the channel dimension; Conv1() represents the convolution operation of the convolution kernel of size 1×1; ReLU represents the ReLU activation function; HardS represents the Hard Sigmoid activation function;

[0446] S3342: Perform element-wise multiplication on Y4 and Y1 (i.e. Figure 12 , we obtain the attention anomaly detection feature map Y5; then we use the convolution kernel of size 3×3 to perform a depth-separable convolution operation on Y5 (i.e. Figure 12 The depthwise separable convolution operation DSC3×3 in the image is used to obtain the attention anomaly detection feature map Y6; then Y6 is activated by the Sigmoid function to obtain the attention anomaly detection feature map Y7; then Y7 and Y5 are element-wise multiplied (i.e. Figure 12 Element-wise multiplication operation ⊙) in , obtains the attention anomaly detection feature map Y8;

[0447] Y8=(Y4⊙Y1)⊙S(DSC3(Y5))

[0448] In the above formula, DSC3() represents the depth-wise separable convolution operation of the convolution kernel of size 3×3; S represents the Sigmoid activation function; ⊙ represents the element-wise multiplication operation;

[0449] S3343: Use the fully connected layer to perform feature mapping on Y8 (i.e. Figure 12 FC module in the FC module) to obtain the attention anomaly detection feature map Y9; then, Y9 is processed by the Sigmoid activation function to obtain the attention anomaly detection feature map Y 10 ; Finally, Y 10 Perform element-wise multiplication with Y8 (i.e. Figure 12, we can get the attention anomaly detection feature map Y 11 ;

[0450] Y 11 =S(FC(Y8))⊙Y8

[0451] In the above formula, S represents the Sigmoid activation function; FC() represents the fully connected layer feature mapping operation; ⊙ represents the element-by-element multiplication operation;

[0452] Example:

[0453] First, input the power equipment detection feature map Y1 of size 28×28×256. First, use the convolution kernel of size 3×3 (step size 1, padding 1), the convolution kernel of size 5×5 (step size 1, padding 2) and the convolution kernel of size 7×7 (step size 1, padding 3) to perform convolution operations on Y1 respectively; then, the three convolution results are spliced in the channel dimension to obtain the attention anomaly detection feature map Y2, the size of Y2 is 28×28×768; then use the convolution kernel of size 1×1 to convolve Y2 to obtain the attention anomaly detection feature map Y3, the size of Y3 is 28×28×256; then Y3 is first processed by the ReLU activation function and then by the Hard Sigmoid activation function to obtain the attention anomaly detection feature map Y4, the size of Y4 is 28×28×256;

[0454] Secondly, Y4 and Y1 are element-wise multiplied to obtain the attention anomaly detection feature map Y5, the size of which is 28×28×256. Then, a 3×3 convolution kernel is used, the step size is set to 1, and the padding is set to 1. A depth-wise separable convolution operation is performed on Y5 to obtain the attention anomaly detection feature map Y6, the size of which is 28×28×256. Then, Y6 is subjected to a Sigmoid activation function to obtain the attention anomaly detection feature map Y7, the size of which is 28×28×256. Then, Y7 and Y5 are element-wise multiplied to obtain the attention anomaly detection feature map Y8, the size of which is 28×28×256.

[0455] Subsequently, the fully connected layer is used to perform feature mapping on Y8 to obtain the attention anomaly detection feature map Y9, the size of which is 28×28×256. After that, Y9 is processed by the Sigmoid activation function to obtain the attention anomaly detection feature map Y 10 , Y 10 The size of Y is 28×28×256; finally, 10 Perform element-wise multiplication with Y8 to obtain the attention anomaly detection feature map Y 11 , Y 11 The size is 28×28×256;

[0456] S335: P 24 Input to the AADM module to obtain the power equipment detection feature map P 25 ;P 27 Input to the AADM module to obtain the power equipment detection feature map P 28 ;P 30 Input to the AADM module to obtain the power equipment detection feature map P 31 ;

[0457] Afterwards, P 25 Through the convolution layer with a convolution kernel size of 1×1 (i.e. Figure 10 Conv2d module in the , and obtain the power equipment detection feature map P 26 ;P 28 Through the convolution layer with a convolution kernel size of 1×1 (i.e. Figure 10 Conv2d module in the , and obtain the power equipment detection feature map P 29 ;P 31 Through the convolution layer with a convolution kernel size of 1×1 (i.e. Figure 10 Conv2d module in the , and obtain the power equipment detection feature map P 32 ;

[0458] Then, P 26 The values at all locations in the spatial dimensions (i.e. height and width) are averaged (i.e. Figure 10 SpatialAvg operation in the image), a two-dimensional vector X1 containing the image fault logits score and the image fault-free logits score is obtained; 29 The values at all locations in the spatial dimensions (i.e. height and width) are averaged (i.e. Figure 10 SpatialAvg operation in the image), and obtain a two-dimensional vector X2 containing the image fault logits score and the image fault-free logits score; P 32 The values at all locations in the spatial dimensions (i.e. height and width) are averaged (i.e. Figure 10 The SpatialAvg operation in the , obtains a two-dimensional vector X3 containing the image fault logits score and the image fault-free logits score; then, the three-level two-dimensional vectors are added element by element (i.e. Figure 10 +), and obtain the comprehensive logits vector X4; finally, the comprehensive logits vector is converted into a probability distribution through the Softmax activation function to obtain a two-dimensional vector P containing the probability of image prediction failure and the probability of image prediction failure-free. out , then according to P out Get the image prediction failure probability And the image prediction failure-free probability

[0459] X4=X1+X2+X3

[0460] P out =Softmax(X4)

[0461]

[0462] In the above formula, Softmax represents the Softmax activation function; + represents the element-by-element addition operation;

[0463] Where: logits is the unnormalized score for calculating class probabilities;

[0464] S33 embodiment:

[0465] The power equipment image P1 in the image sub-dataset obtained by S2 is used as the input of the power equipment image anomaly detection network. The size of P1 is 224×224 pixels and the number of channels is 3. P1 is input into the CBS module to obtain the power equipment detection feature map P2. The size of P2 is 112×112 pixels and the number of channels is 64. Then, P2 is passed through the CBS module to obtain the power equipment detection feature map P3. The size of P3 is 56×56 pixels and the number of channels is 128. After that, P3 is passed through the C2F module to obtain the power equipment detection feature map P4. The size of P4 is 56×56 pixels and the number of channels is 128. Then, P4 is passed through the CBS module to obtain the power equipment detection feature map P5. The size of P5 is 28×28 pixels and the number of channels is 256; P5 is input into the CFAE module to obtain the power equipment detection feature map P6, the size of P6 is 28×28 pixels and the number of channels is 256; P6 is input into the CBS module to obtain the power equipment detection feature map P7, the size of P7 is 14×14 pixels and the number of channels is 512; then, P7 passes through the CFAE module to obtain the power equipment detection feature map P8, the size of P8 is 14×14 pixels and the number of channels is 512; then, P8 passes through the CBS module to obtain the power equipment detection feature map P9, the size of P9 is 7×7 pixels and the number of channels is 1024; then, P9 passes through the CFAE module to obtain the power equipment detection feature map P 10 , P 10 The size of P is 7×7 pixels and the number of channels is 1024; 10 Through the SPPF module, the power equipment detection feature map P is obtained 11 , P 11 The size is 7×7 pixels and the number of channels is 1024;

[0466] Then, P 11Perform upsampling operation to obtain the power equipment detection feature map P 12 , P 12 The size of P is 14×14 pixels and the number of channels is 512. Next, P8 and P 12 Splicing is performed on the channel dimension to obtain the power equipment detection feature map P 13 , P 13 The size of P is 14×14 pixels and the number of channels is 1024; then 13 Through the C2F module, the power equipment detection feature map P is obtained 14 , P 14 The size of P is 14×14 pixels and the number of channels is 512; 14 Through upsampling operation, the power equipment detection feature map P is obtained 15 , P 15 The size of P is 28×28 pixels and the number of channels is 256. Then, P6 and P 15 Splicing is performed on the channel dimension to obtain the power equipment detection feature map P 16 , P 16 The size of P is 28×28 pixels and the number of channels is 512; 16 Through the C2F module, the power equipment detection feature map P is obtained 17 , P 17 The size of P is 28×28 pixels and the number of channels is 256; 17 Through the CBS module, the power equipment detection characteristic map P is obtained 18 , P 18 The size of P is 28×28 pixels and the number of channels is 256; 14 and P 18 Perform channel dimension splicing to obtain the power equipment detection feature map P 19 , P 19 The size of P is 28×28 pixels and the number of channels is 768; then, 19 Through the C2F module, the power equipment detection feature map P is obtained 20 , P 20 The size of P is 28×28 pixels and the number of channels is 256; then, 20 Through the CBS module, the power equipment detection characteristic map P is obtained 21 , P 21 The size of P is 28×28 pixels and the number of channels is 256; 11 and P 21 Splicing is performed on the channel dimension to obtain the power equipment detection feature map P 22 , P 22 The size of P is 28×28 pixels and the number of channels is 1280; Next, 22Through the C2F module, the power equipment detection feature map P is obtained 23 , P 23 The size is 28×28 pixels and the number of channels is 256;

[0467] Next, P 17 Input to the AADM module to obtain the power equipment detection feature map P 24 , P 24 The size of P is 28×28 pixels and the number of channels is 256; 20 Input to the AADM module to obtain the power equipment detection feature map P 27 , P 27 The size of P is 28×28 pixels and the number of channels is 256; 23 Input to the AADM module to obtain the power equipment detection feature map P 30 , P 30 The size of P is 28×28 pixels and the number of channels is 256; then, 24 Input to the AADM module to obtain the power equipment detection feature map P 25 , P 25 The size of P is 28×28 pixels and the number of channels is 256; 27 Input to the AADM module to obtain the power equipment detection feature map P 28 , P 28 The size of P is 28×28 pixels and the number of channels is 256; 30 Input to the AADM module to obtain the power equipment detection feature map P 31 , P 31 The size of P is 28×28 pixels and the number of channels is 256; then, 25 Through the convolution layer with a convolution kernel size of 1×1, the power equipment detection feature map P is obtained 26 , P 26 The size of P is 28×28 pixels and the number of channels is 2; 28 Through the convolution layer with a convolution kernel size of 1×1, the power equipment detection feature map P is obtained 29 , P 29 The size of P is 28×28 pixels and the number of channels is 2; 31 Through the convolution layer with a convolution kernel size of 1×1, the power equipment detection feature map P is obtained 32 , P 32 The size is 28×28 pixels and the number of channels is 2;

[0468] Then, P 26 The values of all positions in the spatial dimension (ie, height and width) are averaged to obtain a two-dimensional vector X1 containing the image fault logits score and the image fault-free logits score;29 The values of all positions in the spatial dimension (ie, height and width) are averaged to obtain a two-dimensional vector X2 containing the image fault logits score and the image fault-free logits score; P 32 The values at all positions in the spatial dimension (i.e., height and width) are averaged to obtain a two-dimensional vector X3 containing the image fault logits scores and the image fault-free logits scores; then, the three-level two-dimensional vectors are added element by element to obtain the comprehensive logits vector X4; finally, the comprehensive logits vector is converted into a probability distribution through the Softmax activation function, and the output image prediction fault probability is And the image prediction failure-free probability

[0469] S34, determining whether the power equipment has a fault: combining the predicted probabilities of each modality, and then performing decision-level fusion using a multimodal weighted fusion (MMWF) algorithm to obtain an overall predicted probability of a fault and an overall predicted probability of no fault; and determining whether the power equipment has a fault based on the probability values; if the overall predicted probability of a fault is greater than the overall predicted probability of no fault, then outputting that the equipment has a fault; otherwise, outputting that the equipment has no fault;

[0470] The present invention designs a multi-modal weighted fusion algorithm MMWF (Multi-Modal Weighted Fusion Algorithm), wherein the overall structure of MMWF is as follows Figure 13 As shown;

[0471] The specific steps of S34 are as follows:

[0472] Input sub-model M1 predicts the fault probability of the audio frequency of the power equipment and audio prediction failure-free probability Sub-model M2 predicts the probability of failure of power equipment vibration and vibration prediction failure-free probability Sub-model M3 predicts the probability of failure of power equipment images And the image prediction failure-free probability First, calculate the mutual information I(M1,M2) between M1 and M2, calculate the mutual information I(M1,M3) between M1 and M3, and calculate the mutual information I(M2,M3) between M2 and M3. Then, adjust the weight of each sub-model according to the mutual information calculation results to obtain the adjusted M1 model weight w adjust (1) M2 model weight w adjust (2) and M3 model weight w adjust(3) Then apply the weighted voting method to calculate the weighted voting probability P of the fault vote (1) and the weighted voting probability P of no fault vote (2); Then the soft voting method is applied to calculate the soft voting probability P of the faulty soft (1) and the probability of fault-free soft voting P soft (2); Finally, the results of weighted fusion weighted voting method and soft voting method are weighted to obtain the final prediction result: the overall predicted fault probability P mmwf (1) and the overall predicted failure-free probability P mmwf (2);

[0473]

[0474]

[0475] P mmwf (y)=α*P vote (y)+(1-α)*P soft (y)

[0476] In the above formula, I(M i ,M j ) indicates the calculation of M i and M j The mutual information between them; y represents the binary classification value of whether there is a fault, where y = 1 means there is a fault and y = 2 means there is no fault; Represents the sub-model M i The predicted probability of category y; P(y) is the prior probability of category y, P(y=1) is equal to the ratio of the number of faulty samples to the total number of samples in the data set, and P(y=2) is equal to the ratio of the number of non-faulty samples to the total number of samples in the data set; w i Represents each sub-model M i The learnable weights; k represents the number of sub-models; w adjust (i) represents the sub-model M after adjustment based on mutual information i Normalized weight of 1(y=y pred,i ) represents the sub-model M i An indicator function indicating whether the prediction is for category y; α represents a learnable parameter used to balance the influence of weighted voting and soft voting;

[0477] The weighted voting method is a multi-model decision fusion strategy that assigns different weights to each sub-model to reflect the reliability differences in their prediction results. The specific process is as follows: First, a weight coefficient is set for each sub-model (such as audio, vibration, and image analysis models) based on the performance of the validation set or prior knowledge. Then, the failure probability output by each model is weighted and summed. Finally, the device status is determined based on the weighted score. For example, if the image model has a higher classification accuracy, it can be given a larger weight (such as 0.5), while the vibration and audio models can be weighted 0.3 and 0.2, respectively. This method emphasizes the contribution of high-confidence models through differentiated weighting and is suitable for scenarios where the performance of each sub-model is significantly uneven.

[0478] The soft voting method is an algorithm that achieves multi-model fusion by directly averaging the probability distributions output by each sub-model. Its core idea is that the predicted probabilities of all sub-models are equally important. The failure and normal probabilities output by each model are arithmetic averaged, and the average probability is ultimately used as the basis for comprehensive judgment. For example, if the three sub-models predict the failure probabilities as 0.7, 0.8, and 0.6, respectively, the fusion result is (0.7 + 0.8 + 0.6) / 3 = 0.7. This method does not require weight setting and emphasizes "collective decision-making." It is suitable for situations where the performance of the sub-models is similar and the prediction results are complementary, and can effectively reduce the impact of random errors of individual models.

[0479] Example:

[0480] Input sub-models M1, M2, and M3 respectively predict the probability of power equipment failure and the predicted probability of no failure Prior probability: P(y=1)=0.5, P(y=2)=0.5 (assuming that the positive and negative samples in the data set are balanced); w i Take 1 / 3 and α as 0.5;

[0481] The first step is to calculate the mutual information between models

[0482]

[0483] Similarly, we can calculate: I(M1,M3)≈0.3829, I(M2,M3)≈0.2819

[0484] The second step is to calculate the adjusted model weights

[0485]

[0486] By the same calculation, we can get:

[0487] w adjust (2)≈0.3263

[0488] wadjust (3)≈0.3276

[0489] Step 3: Apply weighted voting

[0490] P vote (1) = w adjust (1)*1(y=1)+w adjust (3)*1(y=1)

[0491] =0.3461+0.3276=0.6737

[0492] P vote (2) = w adjust (2)*1(y=2)=0.3263

[0493] Step 4: Apply soft voting

[0494]

[0495] Step 5: Calculate the MMWF prediction results

[0496] Probability of failure:

[0497] P mmwf (1) = α*P vote (y=1)+(1-α)*P soft (y=1)

[0498] =0.5*0.6737+0.5*0.6004=0.6371

[0499] Failure-free probability:

[0500] P mmwf (2) = α*P vote (y=2)+(1-α)*P soft (y=2)=0.5*0.3263+0.5*0.3996=0.3630;

[0501] S4: Divide the preprocessed power equipment multimodal dataset, and then train and verify the power equipment anomaly detection model;

[0502] The specific steps of S4 are as follows:

[0503] S41: Divide the pre-processed multimodal dataset of power equipment in S2 into a training set, a validation set, and a test set according to a certain ratio, ensuring that the samples in the training set, validation set, and test set are independent and non-overlapping; during the division process, ensure that the category distribution of each subset is consistent with the overall dataset to avoid the negative impact of data imbalance on model training;

[0504] S42: A dynamically adjusted hyperparameter scheme is adopted: the basic training cycle is set to a certain number of rounds, and a progressive batch expansion strategy is implemented, that is, the initial batch size is set to a small value, and the batch size is gradually increased after each training cycle. The learning rate scheduling adopts the cyclic cosine annealing algorithm, starting from a low baseline value, and setting the cycle length according to the training requirements, so that the learning rate decreases smoothly during the training process, effectively improving the convergence of the model. The optimizer uses NAdam and sets the gradient normalization constraint. By limiting the L2 norm of the parameter update, the gradient is prevented from being too large, ensuring the stability of the training process.

[0505] S43: During the training phase, the input samples in each batch are a five-tuple, consisting of: normalized audio data from the one-dimensional sound sub-dataset, mel-spectrograms from the two-dimensional sound sub-dataset, normalized vibration signals from the one-dimensional vibration sub-dataset, Gram angular field images from the two-dimensional vibration sub-dataset, and images of power equipment from the image sub-dataset. Using these five-tuple samples, the model can learn the correlations between different modalities. During feature extraction, a cross-modal contrast loss function is used to optimize the model, effectively aligning data from different modalities in the feature space and improving multimodal learning capabilities.

[0506] S44: During the validation phase, a multi-dimensional evaluation protocol is used, with full validation performed every several batches. The validation process includes several test modes: standard data inference, noise interference testing, and modal loss testing. An adaptive early stopping mechanism is also implemented: when the coefficient of variation of the main evaluation metric remains below a preset value for several consecutive validation runs, training is automatically terminated to ensure model convergence and stability.

[0507] S45: During the testing phase, the model will undergo several rigorous tests on the test set, including cross-device generalization assessment, real-time stress testing, model stability and robustness, and performance evaluation under data noise interference. These tests ensure that the model can maintain high reliability and accuracy in practical applications and meet the actual needs of power equipment anomaly detection.

[0508] Example:

[0509] S41 divides the preprocessed multimodal dataset of power equipment in S2 into a training set, a validation set, and a test set in a ratio of 7:2:1, resulting in a training set containing 7,000 data items, a validation set containing 2,000 data items, and a test set containing 1,000 data items. Meanwhile, during the division process, the category distribution of each subset is ensured to be consistent with that of the entire dataset.

[0510] S42 uses a dynamically adjusted hyperparameter scheme: the base training cycle is set to 120 epochs; a progressive batch expansion strategy is implemented, with an initial batch size of 32, which is doubled after every 30 training epochs; the learning rate scheduling uses the cyclic cosine annealing algorithm, with a baseline value set to 0.002 and a cycle length of 40 training epochs; the optimizer uses NAdam, and a gradient normalization constraint is set, with the L2 norm of parameter updates capped at 2.0;

[0511] During the training phase of S43, the input samples in each batch are a five-tuple, including: normalized audio data from the one-dimensional sound sub-dataset, Mel-spectrograms from the two-dimensional sound sub-dataset, normalized vibration signals from the one-dimensional vibration sub-dataset, Gram angular field images from the two-dimensional vibration sub-dataset, and images of power equipment from the image sub-dataset. During the feature extraction process, a cross-modal contrast loss function is used to optimize the model so that data from different modalities can be effectively aligned in the feature space.

[0512] During the validation phase, S44 uses a multi-dimensional evaluation protocol, performing a comprehensive validation every five batches. The validation process includes three test modes: standard data inference (for basic performance evaluation), noise interference testing (adding Gaussian noise to the data to simulate environmental noise), and modal absence testing (randomly blocking some modal inputs). An adaptive early stopping mechanism is also implemented: training is automatically terminated when the coefficient of variation of the main evaluation metric remains below 5% for 15 consecutive validation runs, ensuring model convergence stability.

[0513] During the S45 testing phase, the model underwent five rigorous tests on a 1,000-item test set, including but not limited to: cross-device generalization evaluation, real-time stress testing, model stability and robustness, and performance evaluation under data noise interference. This ensures that the model can maintain high reliability and accuracy in practical applications and meet the actual needs of power equipment anomaly detection.

[0514] S5: Apply the trained power equipment anomaly detection model to detect power equipment;

[0515] The specific steps of S5 are as follows:

[0516] S51: Input the collected audio data, vibration signals, and images of the power equipment into a power equipment anomaly detection model to obtain an overall predicted probability of failure and an overall predicted probability of no failure;

[0517] S52: Then a judgment is made. If the overall predicted probability of failure is greater than the overall predicted probability of no failure, the output device is faulty; otherwise, the output device is not faulty.

[0518] In summary, the present invention extracts multimodal abnormal features from audio data, vibration signals, and images of power equipment and performs multimodal fusion to detect whether there is a fault in power equipment. Specifically, the present invention includes:

[0519] (1) A Mel-spectrogram feature extraction network (MSFE-Net) was designed to extract the frequency domain features of audio data. BiLSTM was used to extract the time domain features of audio data. The frequency domain features and time domain features of the audio data were then fused to fully capture the complex information of the audio data. A spatial adaptive pooling module and a local-global fusion attention module were designed within MSFE-Net to retain and enhance the abnormal features in the audio data.

[0520] (2) A Gram angular field feature extraction network (GAFE-Net) was designed to extract the spatial features of vibration signals. BiGRU was used to extract the temporal features of vibration signals. The spatial and temporal features of vibration signals were then fused to extract abnormal information of vibration signals in complex environments. A dynamic anomaly perception channel optimization module and a multi-level adaptive fusion module were designed within GAFE-Net to improve the model's ability to identify abnormal vibration signals in complex environments.

[0521] (3) An electric power equipment image anomaly detection network ADEI-Net was designed to process electric power equipment images and output the image prediction probability of fault and the image prediction probability of no fault of the electric power equipment; a channel feature adaptive enhancement module and an attention anomaly detection module were designed within ADEI-Net to enhance the feature extraction capability of abnormal areas in the image and improve the detection accuracy of image anomalies.

[0522] (4) A multimodal weighted fusion algorithm was designed, which introduced mutual information into the weighted voting method and soft voting method. It can automatically adjust the weight according to the contribution of each modality in the power equipment anomaly detection task, ensuring that the fusion result can fully utilize the information of each modality, thereby improving the comprehensive effect of power equipment anomaly detection.

[0523] Although the present invention has been described above with reference to embodiments, various modifications may be made thereto and equivalent components may be substituted without departing from the scope of the present invention. In particular, as long as there are no structural conflicts, the various features of the embodiments disclosed herein may be combined with each other in any manner, and the omission of an exhaustive description of such combinations in this specification is solely for the sake of space and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for detecting abnormalities in power equipment based on multimodal deep learning, characterized in that: The specific steps are as follows: S1: Build an original multimodal dataset of power equipment covering audio data, vibration signals and images of power equipment; S2: preprocessing the original power equipment multimodal dataset to obtain a preprocessed power equipment multimodal dataset; S3: Build a power equipment anomaly detection model based on multimodal deep learning; S4: Divide the preprocessed power equipment multimodal dataset, and then train and verify the power equipment anomaly detection model; S5: Use the trained power equipment anomaly detection model to detect power equipment.

2. The method for detecting abnormalities in power equipment based on multimodal deep learning according to claim 1, characterized in that: The specific steps of S1 are as follows: S11: First, use a sound sensor to collect audio data generated by the power equipment during operation and save it as an audio file; then, use a vibration sensor to collect vibration signals of the power equipment and save it as a vibration data file; then, use a high-definition camera to capture the working area of the power equipment, record images of the power equipment covering key components, and save them as image files; S12: Label the collected audio files, vibration data files and image files; determine whether the current equipment has a fault from the audio data in the audio file, and then label the audio data as faulty or not; determine whether the current equipment has a fault from the vibration signal in the vibration data file, and then label the vibration signal as faulty or not; determine whether the current equipment has a fault from the power equipment image in the image file, and then use the image labeling tool to label the power equipment image as faulty or not.

3. The method for detecting abnormalities in power equipment based on multimodal deep learning according to claim 1, characterized in that: The specific steps of S2 are as follows: S21: normalizing the audio data in the audio file, that is, unifying the sampling frequency and sampling duration so that the audio data in the audio file have the same sampling interval, thereby obtaining a one-dimensional sound sub-dataset; S22: Processing the audio data in the audio file into a Mel-frequency spectrum using Mel-frequency cepstral coefficients to obtain a two-dimensional sound sub-dataset; S23: normalizing the vibration signal in the vibration data file so that its amplitude is within a uniform range, and then dividing the vibration data into multiple time periods using a sliding window method, where each time period contains vibration data of uniform length, to obtain a one-dimensional vibration sub-dataset; S24: converting the vibration signal in the vibration data file into a Gram angle field diagram to obtain a two-dimensional vibration sub-dataset; S25: resizing the power equipment images in the image file to a uniform size, then using data enhancement technology to generate multiple enhanced images and adding them to the dataset to obtain an image sub-dataset; S26: Performing a time alignment operation: For each sampling time point, extracting data samples at the corresponding time from all sub-datasets, including the normalized audio data in the one-dimensional sound sub-dataset, the Mel-spectrogram in the two-dimensional sound sub-dataset, the normalized vibration signal in the one-dimensional vibration sub-dataset, the Gram angle field image in the two-dimensional vibration sub-dataset, and the power equipment image in the image sub-dataset, and combining them to form a five-tuple sample; The process achieves cross-modal synchronization through timestamp matching to ensure that the data samples of all sub-datasets are aligned in the time dimension; this alignment operation is repeated for all available time points to obtain the preprocessed multimodal dataset of power equipment.

4. The method for detecting abnormalities in power equipment based on multimodal deep learning according to claim 1, characterized in that: The specific steps of S3 are as follows: S31, anomaly detection based on power equipment audio data: First, the mel-spectrogram in the two-dimensional sound sub-dataset of S2 is used to extract the frequency domain features of the audio data. Then, the normalized audio data in the one-dimensional sound sub-dataset of S2 is used to extract the time domain features of the audio data through BiLSTM. The two results are then fused at the feature layer using time-frequency collaborative features to obtain an audio feature map of the power equipment audio data. The audio feature map of the power equipment audio data is then classified using an MLP classifier. Finally, the classification results are converted into audio prediction probabilities of fault presence and non-fault presence using a Softmax activation function. S32, anomaly detection based on vibration signals of power equipment: First, the Gram angle field map in the two-dimensional vibration sub-dataset of S2 is used to extract the spatial features of the vibration signal through GAFE-Net. Then, the normalized vibration signal in the one-dimensional vibration sub-dataset of S2 is used to extract the temporal features of the vibration signal through BiGRU. Subsequently, the two results are combined into a time series feature fusion at the feature layer to obtain the vibration feature map of the power equipment vibration signal. The vibration feature map of the power equipment vibration signal is then classified using an MLP classifier. Finally, the classification results are converted into the vibration prediction fault probability and the vibration prediction non-fault probability using a Softmax activation function. S33, anomaly detection based on power equipment images: For power equipment images in image files, ADEI-Net is used to perform target detection on the power equipment images, extract visual features of the equipment, and determine the anomalies of the images. The predicted probability of the images being faulty and the predicted probability of being fault-free are output. S34, determine whether the power equipment has a fault: combine the predicted probabilities of each mode, and then perform decision-level fusion through a multi-modal weighted fusion algorithm to obtain the overall predicted probability of fault and the overall predicted probability of no fault; and determine whether the power equipment has a fault based on the probability values; if the overall predicted probability of fault is greater than the overall predicted probability of no fault, then output that the equipment has a fault; otherwise, output that the equipment has no fault.

5. The method for detecting abnormalities in power equipment based on multimodal deep learning according to claim 4, characterized in that: The specific steps of S31 are as follows: S311: The Mel spectrum graph in the two-dimensional sound sub-dataset is passed through the Mel spectrum graph feature extraction network MSFE-Net to extract the frequency domain features of the audio data and obtain the Mel high-level feature graph F 19 ; S312: Extracting time domain features of audio data: Input the normalized audio data in the one-dimensional sound sub-dataset into a bidirectional long short-term memory network (BiLSTM); First, the forward LSTM layer in BiLSTM processes the audio data from left to right according to the time step, gradually updating the forward hidden state to capture the forward dependency of the audio data in the time series; at the same time, the reverse LSTM layer in BiLSTM processes the audio data from right to left according to the time step, updating the reverse hidden state to capture the backward dependency of the audio data; Next, the forward and reverse hidden states are concatenated at each time step to obtain the bidirectional features of the time step; finally, a global average pooling operation is performed on the bidirectional features of all time steps to obtain the time domain feature map E1; S313: F 19 The audio feature map of the power equipment audio data is obtained by splicing E1 and E2 in the feature dimension. Then, the audio feature map of the power equipment audio data is classified by the MLP classifier to obtain a two-dimensional vector containing the audio fault logits score and the audio fault-free logits score. Finally, the two-dimensional vector is normalized by the Softmax activation function to obtain the audio prediction fault probability. and audio prediction failure-free probability The specific steps of S311 are as follows: S3111: Denote any mel-spectrogram in the two-dimensional sound sub-dataset as F1, input it into a convolutional layer with 64 3×3 convolution kernels and a stride of 1, and then pass it through the ReLU activation function and batch normalization to obtain the mel shallow feature map F2; then, input F2 into a convolutional layer with 64 3×3 convolution kernels and a stride of 1, and then pass it through the ReLU activation function and batch normalization to obtain the mel shallow feature map F3; S3112: Input F3 into the SAPM module to obtain the Mel shallow feature map F4; where F3 is denoted as A1; S3113: Input F4 to a convolutional layer with 128 3×3 convolution kernels and a stride of 1, and then pass through the ReLU activation function and batch normalization to obtain the Mel shallow feature map F5; then input F5 to a convolutional layer with 128 3×3 convolution kernels and a stride of 1, and then pass through the ReLU activation function and batch normalization to obtain the Mel shallow feature map F6; then, input F6 to the SAPM module to obtain the Mel shallow feature map F7; S3114: Input F7 to a convolutional layer with 256 3×3 convolution kernels and a stride of 1, and then pass through the ReLU activation function and batch normalization to obtain the Mel middle layer feature map F8; then input F8 to a convolutional layer with 256 3×3 convolution kernels and a stride of 1, and then pass through the ReLU activation function and batch normalization to obtain the Mel middle layer feature map F9; then, input F9 to the SAPM module to obtain the Mel middle layer feature map F 10 ; S3115: F 10 The input is sent to a convolutional layer with 512 3×3 convolution kernels and a stride of 1, and then processed by ReLU activation function and batch normalization to obtain the Mel high-level feature map F 11 ; Then F 11 The input is sent to a convolutional layer with 512 3×3 convolution kernels and a stride of 1, and then processed by ReLU activation function and batch normalization to obtain the Mel high-level feature map F 12 ; S3116: F 12 Input to the LGFA module to obtain the Mel high-level feature map F 13 ; S3117: F 13 Input to the SAPM module to obtain the Mel high-level feature map F 14 ; Then F 14 The input is sent to a convolutional layer with 512 3×3 convolution kernels and a stride of 1, and then processed by ReLU activation function and batch normalization to obtain the Mel high-level feature map F 15 ; Then, F 15 The input is sent to a convolutional layer with 512 3×3 convolution kernels and a stride of 1, and then processed by ReLU activation function and batch normalization to obtain the Mel high-level feature map F 16 ; Then, F 16 Input to the LGFA module to obtain the Mel high-level feature map F 17 ; Next, F 17 Input to the SAPM module to obtain the Mel high-level feature map F 18 ; Finally, F 18 Input to the fully connected layer, obtain and output Mel high-level feature map F 19 ; The specific steps of S3112 are as follows: S31121: In the first branch, calculate A along the vertical axis 1_p The mean of the vertical mean vector μ p,h ; In the second branch, calculate A along the vertical axis 1_p The variance of the vertical variance vector m p,h =[μ p,h [1],m p,h [2],…,m p,h [h]] In the above formula, A 1_p (i,j) represents the 1_p The eigenvalue of the (i, j)th position in ; S31122: In the third branch, according to A 1_p The output height H is calculated from the height H 2_p ; S31123: In the fourth branch, A 1_p Do any operation; in the fifth branch, calculate A along the horizontal axis 1_p The mean of the level vector μ p,w , in the sixth branch, calculate A along the horizontal axis 1_p The variance of the horizontal variance vector m p,w =[μ p,w [1],m p,w [2],…,m p,w [w]] In the above formula, A 1_p (i,j) represents the 1_p The eigenvalue of the (i, j)th position in ; S31124: In the seventh branch, according to A 1_p The width W is calculated to get the output width W 2_p ; S31125: Based on variance and Calculate and get the variance factor Based on μ p,h Perform variance guided fusion calculation to obtain the mean μ h , then according to Will Perform variance guided fusion calculation to obtain variance Then according to μ p,w Perform variance guided fusion calculation to obtain the mean μ w , then according to Will Perform variance guided fusion calculation to obtain the mean Then, according to μ h and Calculate and get the vertical pooling window size k h ; According to μ w and Calculate and get the horizontal pooling window size k w ; In the above formula, max() represents the maximum value operation; mean() represents the average value operation; round() means rounding a number to the nearest integer; k min is the size of the minimum pooling window, which is used to avoid the pooling window being too small; ε is a constant used to prevent instability caused by the mean being close to zero or the variance being too small; S31126: According to k h and H 2_p Calculate the vertical step length s h ; Then, according to k w and W 2_p Calculate the horizontal step size s w ; In the above formula, round() means rounding the number to the nearest integer; S31127: According to A 1_p Height H, output height H 2_p , vertical pooling window size k h and vertical step length s h , calculate the vertical padding h According to A 1_p Width W, output width W 2_p , horizontal pooling window size k w and horizontal step length s w , calculate the horizontal padding w ; Next, for A 1_p Perform zero padding on all four sides in sequence: first in A 1_p Add padding to the left w Zero fill the column and then add padding on the right w Zero fill the columns, then add padding on top h Zero padding the rows, then adding padding at the bottom h The row is zero-filled; the final size is (H+2*padding h )×(W+2*padding w )’s four-side zero-filled feature map; Finally, according to the calculated pooling window size k h 、k w and step length s h 、s w , perform the maximum pooling operation on the feature map with zero padding on all four sides to obtain a single-channel feature map A of size H / 2×W / 2 2_p ; The specific steps of S3116 are as follows: S31161: In the first branch, no operation is performed on B1; in the second branch, B1 is first globally average pooled to obtain the local attention feature map B2; a fully connected layer is used to reduce the dimension of B2 to obtain the local attention feature map B3; B3 is then activated by the ReLU function to obtain the local attention feature map B4; another fully connected layer is then used to increase the dimension of B4 to obtain the local attention feature map B5, so that the number of channels of B5 and B2 are consistent; B5 is then activated by the Sigmoid function to obtain the local attention feature map B6; B6 is then element-wise multiplied by B1 to obtain the local attention feature map B7; Subsequently, B7 is first globally average pooled, and the pooling result is transformed by the multi-layer perceptron, and the feature transformation result is passed through the Sigmoid activation function to obtain the local attention feature map B8; Finally, B7 and B8 are element-wise multiplied to obtain the local attention feature map B9; B7=B1⊙S(FC2(ReLU(FC1(GAP(B1))))) B9=B7⊙S(MLP(GAP(B7))) In the above formula, S represents the Sigmoid activation function; GAP() represents the Global Avg Pool operation; ReLU represents the ReLU activation function; MLP() represents the multi-layer perceptron processing; FC1() represents the dimensionality reduction processing through the fully connected layer; FC2() represents the dimensionality increase processing through the fully connected layer; ⊙ represents the element-by-element multiplication operation; S31162: In the third branch, the weight matrix W v Multiply it with B1 to get the value matrix V; in the fourth branch, the weight matrix W q Multiply it with B1 to get the query matrix Q; in the fifth branch, the weight matrix W k Multiply it with B1 to get the key matrix K; then calculate the dot product of Q and K and pass it through the Softmax activation function to get the attention score matrix S; then multiply S with V to get the global attention feature map B 10 ; Then use the convolution kernel of size 1×1 to B 10 Perform convolution operation to obtain the global attention feature map B 11 ; Then, for B 11 Perform global average pooling, then use a multi-layer perceptron to transform the pooling result, and then pass the feature transformation result through the Sigmoid activation function to obtain the global attention feature map B 12 ; Then, B 11 With B 12 Perform element-by-element multiplication to obtain the global attention feature map B 13 ; V=W v ×B1 Q=W q ×B1 K=W k ×B1 B 12 =S(MLP(GAP(B 11 ))) B 13 =B 11 ⊙B 12 In the above formula, S represents the Sigmoid activation function; Softmax represents the Softmax activation function; GAP() represents the GlobalAvg Pool operation; MLP() represents the multi-layer perceptron processing; the He initialization method is used to initialize the weight matrix W. q 、W k and W v Initialize, W q 、W k and W v The dimensions of Where C' is the number of channels after compression; × represents matrix multiplication; ⊙ represents element-by-element multiplication operation; + represents element-by-element addition operation; S31163: Finally, B9 and B 13 Perform element-by-element addition to obtain and output the fused attention feature map B 14 .

6. The method for detecting abnormalities in power equipment based on multimodal deep learning according to claim 4, characterized in that: The specific steps of S32 are as follows: S321: The Gram angular field map in the two-dimensional vibration sub-dataset is passed through the Gram angular field map feature extraction network GAFE-Net to extract the spatial features of the vibration signal and obtain the Gram extraction feature map G 15 ; S322: Extracting the temporal features of the vibration signal: The normalized vibration signal in the one-dimensional vibration sub-dataset is input into a bidirectional gated recurrent unit (BiGRU). First, the forward GRU layer in the BiGRU processes the vibration signal from left to right in chronological order, and gradually updates the forward hidden state using a gating mechanism consisting only of an update gate and a reset gate to capture the forward temporal pattern of the vibration signal. Simultaneously, the reverse GRU layer processes the signal inversely from right to left, controlling the flow of information through gating to capture the reverse dependency of the vibration signal. Then, at each time step, the forward and reverse hidden states are concatenated according to the channel dimension to form a fused feature containing bidirectional temporal information. Finally, the fused features of all time steps are globally averaged and pooled to obtain a temporal feature map N1. S323: G 15 and N1 in the feature dimension to obtain the vibration feature map of the power equipment vibration signal; then, the vibration feature map of the power equipment vibration signal is classified by the MLP classifier to obtain a two-dimensional vector containing the vibration fault logits score and the vibration fault-free logits score; finally, the two-dimensional vector is normalized by the Softmax activation function to obtain the vibration prediction fault probability and vibration prediction failure-free probability The specific steps of S321 are as follows: S3211: Denote any Gram angular field map in the two-dimensional vibrator dataset as G1, input it into a convolution layer with a convolution kernel size of 4×4 and a stride of 4, and then perform layer normalization to obtain a Gram extraction feature map G2; then, pass G2 through three Bottleneck modules in succession to obtain a Gram extraction feature map G3; then, pass G3 through a Downsample module to reduce its height and width by half while keeping the number of channels unchanged, to obtain a Gram extraction feature map G4. The specific operation is as follows: first, pass G3 through layer normalization, and then pass the layer normalization result through a convolution layer with a convolution kernel size of 2×2 and a stride of 2 to obtain G4; then, pass G4 through five Bottleneck modules in succession to obtain a Gram extraction feature map G5; S3212: Input G5 into the DAACO module to obtain the Gram extraction feature map G6; S3213: Input G6 into the Downsample module to reduce its height and width by half while keeping the number of channels unchanged, thereby obtaining the Grammar extraction feature map G7. Then, G7 is passed through seven Bottleneck modules in succession to obtain the Grammar extraction feature map G8. Finally, G8 is passed through the DAACO module to obtain the Grammar extraction feature map G9. S3214: Input G9 into the MLAF module to obtain the Gram extraction feature map G 10 ; S3215: G 10 Input it to the Downsample module to reduce its height and width by half, while keeping the number of channels unchanged, and obtain the Gram extraction feature map G 11 ; Then, G 11 Continuously pass through 5 Bottleneck modules to obtain the Gram extraction feature map G 12 ; Then, G 12 Through the MLAF module, we get the Gram extraction feature map G 13 ; Then, G 13 Perform global average pooling to obtain the Gram extraction feature map G 14 ; Finally, G 14 Perform layer normalization to obtain and output the Gram extraction feature map G 15 ; The specific steps of S3212 are as follows: S32121: In the first branch, no operation is performed on L1; in the second branch, global average pooling is used to pool L1 to obtain the channel optimized feature map L2; in the third branch, a pooling window of size 3×3 and a dynamic step size s are first used. D Perform an average pooling operation on L1, and then perform a global average pooling operation on the average pooling result to obtain the channel optimized feature map L3; in the fourth branch, first use a pooling window of size 5×5 and a dynamic step size s D Perform an average pooling operation on L1, and then perform a global average pooling operation on the average pooling result to obtain the channel optimized feature map L4; L2=GAP(L1) L3=GAP(AvgPool3(L1)) L4=GAP(AvgPool5(L1)) In the above formula, It is a hyperparameter; max() indicates the maximum value operation; GAP() indicates the Global Avg Pool operation; AvgPool3() indicates that the pooling window size is 3×3 and the step size is s D Average pooling operation; AvgPool5() means the pooling window size is 5×5 and the step size is s D Average pooling operation; S32122: Set the channel sparse parameter W S1 Multiply by L2 to get the channel optimized feature map L5; the channel sparse parameter W S2 Multiply it with L3 to get the channel optimized feature map L7; set the channel sparse parameter W S3 Multiply it with L4 to get the channel optimized feature map L9; then perform element-by-element addition operation on L5, L7 and L9 to get the channel optimized feature map L 11 ; Then L 11 After the Sigmoid activation function, the channel optimized feature map L is obtained 12 ; Then L 12 Perform element-by-element multiplication with L1 to obtain the channel optimized feature map L 13 ; L 13 =S(W S1 ×L2+W S2 ×L3+W S3 ×L4)⊙L1 In the above formula, the Xavier initialization method is used to initialize the channel sparse parameter W S1 、W S2 and W S3 Initialize; S represents the Sigmoid activation function; × represents matrix multiplication; ⊙ represents element-by-element multiplication operation; + represents element-by-element addition operation; S32123: Set the abnormal channel parameter W A1 Multiply by L2 to get the channel optimization feature map L6; the abnormal channel parameter W A2 Multiply it by L3 to get the channel optimization feature map L8; the abnormal channel parameter W A3 Multiply by L4 to get the channel optimized feature map L 10 ; Then L6, L8 and L 10 Perform element-by-element addition to obtain the channel optimized feature map L 14 ; Then L 14 After the ReLU activation function, the channel optimized feature map L is obtained 15 ; L 15 =ReLU(W A1 ×L6+W A2 ×L8+W A3 ×L 10 ) In the above formula, Gaussian distribution is used Initialization method, abnormal channel parameter W A1 、W A2 and W A3 Initialize; ReLU represents the ReLU activation function; × represents matrix multiplication; + represents element-by-element addition operation; S32124: L 13 With L 15 Perform element-by-element multiplication to obtain the channel optimized feature map L 16 ; Afterwards, the channel attention parameter W D With L 16 Perform multiplication operation to obtain the channel optimized feature map L 17 ; Then L 17 After the Softmax activation function, the channel optimized feature map L is obtained 18 ; Finally, L 16 With L 18 Perform element-by-element multiplication to obtain and output the channel optimized feature map L 19 ; L 19 =L 16 ⊙Softmax(W D ×(L 13 ⊙L 15 )) In the above formula, Gaussian distribution is used Initialization method, channel attention parameter W D Initialize; Softmax represents the Softmax activation function; × represents matrix multiplication; ⊙ represents element-by-element multiplication operation; The specific steps of S3214 are as follows: S32141: In the first branch, no operation is performed on T1; in the second branch, no operation is performed on T1; in the third branch, wavelet transform is first performed on T1 to extract high-frequency abnormal features, and a multi-level fusion feature map T2 is obtained; then a convolution operation is performed on T2 with a convolution kernel of size 1×1 to obtain a multi-level fusion feature map T3; then T3 is activated by the Sigmoid function to obtain a multi-level fusion feature map T4: T4 = S(Conv1(DWT(T1))) In the above formula, DWT() represents the wavelet transform operation; Conv1() represents the convolution operation of the convolution kernel with a size of 1×1; S represents the Sigmoid activation function; S32142: In the fourth branch, a depth-separable convolution operation is performed on T1 using a convolution kernel of size 1×1 to obtain a multi-level fusion feature map T5; in the fifth branch, a depth-separable convolution operation is performed on T1 using a convolution kernel of size 3×3 to obtain a multi-level fusion feature map T6; in the sixth branch, a depth-separable convolution operation is performed on T1 using a convolution kernel of size 5×5 to obtain a multi-level fusion feature map T7; in the seventh branch, a depth-separable convolution operation is performed on T1 using a convolution kernel of size 7×7 to obtain a multi-level fusion feature map T8; After that, T5, T6, T7 and T8 are added element by element to obtain the multi-level fusion feature map T9; then T9 is activated by the Sigmoid function to obtain the multi-level fusion feature map T 10 : T 10 =S(DSC1(T1)+DSC3(T1)+DSC5(T1)+DSC7(T1)) In the above formula, DSC1() represents the depth-separable convolution operation of the convolution kernel of size 1×1; DSC3() represents the depth-separable convolution operation of the convolution kernel of size 3×3; DSC5() represents the depth-separable convolution operation of the convolution kernel of size 5×5; DSC7() represents the depth-separable convolution operation of the convolution kernel of size 7×7; + represents the element-by-element addition operation; S represents the Sigmoid activation function; S32143: In the eighth branch, an average pooling operation is performed on T1 in the channel dimension. The specific operation is: traverse each spatial position (i, j) of T1 (where i∈[1,H], j∈[1,W]), and calculate the average value of its position in all channels; in the ninth branch, a maximum pooling operation is performed on T1 in the channel dimension. The specific operation is: traverse each spatial position (i, j) of T1 (where i∈[1,H], j∈[1,W]), and calculate the maximum value of its position in all channels; then the results of the eighth and ninth branches are spliced in the channel dimension to obtain a multi-level fusion feature map T 11 ; Then, a convolution kernel of size 1×1 is applied to T 11 Perform convolution operation to obtain a multi-level fusion feature map T 12 ; Then use the convolution kernel of size 7×7 to T 12 Perform convolution operation to obtain a multi-level fusion feature map T 13 ; Use a convolution kernel of size 3×3 and a dilation rate of 2 to T 12 Perform convolution operation to obtain a multi-level fusion feature map T 14 ; Then T 13 and T 14 Perform element-by-element addition to obtain a multi-level fusion feature map T 15 ; Then T 15 After the Sigmoid activation function, the multi-level fusion feature map T is obtained 16 : T 15 =S(Conv7(T 12 )+Conv3_2(T 12 )) In the above formula, T1(i,j,k) represents the eigenvalue of T1 at the spatial position (i,j) and the kth channel; max k∈[1,C] () indicates the operation of calculating the maximum value of each spatial position in all channels; Concat() indicates the concatenation operation in the channel dimension; Conv7() indicates the convolution operation of a convolution kernel of size 7×7; Conv3_2() indicates the convolution operation of a convolution kernel of size 3×3 and a void rate of 2; S indicates the Sigmoid activation function; + indicates the element-by-element addition operation; S32144: T1, T4, T 10 and T 16 Perform element-by-element multiplication to obtain a multi-level fusion feature map T 17 ; Then use the convolution kernel of size 1×1 to T 17 Perform convolution operation to obtain a multi-level fusion feature map T 18 ; Then T 18 After the Softmax activation function, the abnormal mask M is obtained a ; Afterwards, M a Perform two small branch calculations; in the first small branch, calculate M a The reverse exception mask of 1-a ; Then T1 and M 1-a Perform element-by-element multiplication to obtain a multi-level fusion feature map T 19 ; In the second branch, T 17 and M a Perform element-by-element multiplication to obtain a multi-level fusion feature map T 20 ; Finally, T 19 and T 20 Perform element-by-element addition to obtain and output a multi-level fusion feature map T 21 ; M a =Softmax(Conv1(T1⊙T4⊙T 10 ⊙T 16 )) M 1-a =1-M a T 21 =T1⊙M 1-a +M a ⊙T 17 In the above formula, Conv1() represents the convolution operation of the convolution kernel with a size of 1×1; Softmax represents the Softmax activation function; ⊙ represents the element-by-element multiplication operation; + represents the element-by-element addition operation.

7. The method for detecting abnormalities in power equipment based on multimodal deep learning according to claim 4, characterized in that: The specific steps of S33 are as follows: S331: Denote any power equipment image in the image sub-dataset as P1 and input it into the CBS module to obtain a power equipment detection feature map P2; then, pass P2 through the CBS module to obtain a power equipment detection feature map P3; then, pass P3 through the C2F module to obtain a power equipment detection feature map P4; then, pass P4 through the CBS module to obtain a power equipment detection feature map P5; S332: Input P5 into the CFAE module to obtain the power equipment detection characteristic diagram P6; S333: Input P6 into the CBS module to obtain the power equipment detection feature map P7; then, pass P7 through the CFAE module to obtain the power equipment detection feature map P8; then, pass P8 through the CBS module to obtain the power equipment detection feature map P9; then, pass P9 through the CFAE module to obtain the power equipment detection feature map P 10 ; Then P 10 Through the SPPF module, the power equipment detection feature map P is obtained 11 ; Then, P 11 Perform upsampling operation to obtain the power equipment detection feature map P 12 ; Next, P8 and P 12 Splicing is performed on the channel dimension to obtain the power equipment detection feature map P 13 ; Then P 13 Through the C2F module, the power equipment detection feature map P is obtained 14 ; Then P 14 Perform upsampling operation to obtain the power equipment detection feature map P 15 ; Next, P6 and P 15 Splicing is performed on the channel dimension to obtain the power equipment detection feature map P 16 ; Then P 16 Through the C2F module, the power equipment detection feature map P is obtained 17 ; Then P 17 Through the CBS module, the power equipment detection characteristic map P is obtained 18 ; Next, P 14 and P 18 Splicing is performed on the channel dimension to obtain the power equipment detection feature map P 19 ; Then, P 19 Through the C2F module, the power equipment detection feature map P is obtained 20 ; Then, P 20 Through the CBS module, the power equipment detection characteristic map P is obtained 21 ; Then P 11 and P 21 Splicing is performed on the channel dimension to obtain the power equipment detection feature map P 22 ; Next, P 22 Through the C2F module, the power equipment detection feature map P is obtained 23 ; S334: P 17 Input to the AADM module to obtain the power equipment detection feature map P 24 ;P 20 Input to the AADM module to obtain the power equipment detection feature map P 27 ;P 23 Input to the AADM module to obtain the power equipment detection feature map P 30 ; S335: P 24 Input to the AADM module to obtain the power equipment detection feature map P 25 ;P 27 Input to the AADM module to obtain the power equipment detection feature map P 28 ;P 30 Input to the AADM module to obtain the power equipment detection feature map P 31 ; Afterwards, P 25 Through the convolution layer with a convolution kernel size of 1×1, the power equipment detection feature map P is obtained 26 ;P 28 Through the convolution layer with a convolution kernel size of 1×1, the power equipment detection feature map P is obtained 29 ;P 31 Through the convolution layer with a convolution kernel size of 1×1, the power equipment detection feature map P is obtained 32 ; Then, P 26 The values of all positions in the spatial dimension are averaged to obtain a two-dimensional vector X1 containing the image fault logits score and the image fault-free logits score; 29 The values at all positions in the spatial dimension are averaged to obtain a two-dimensional vector X2 containing the image fault logits score and the image fault-free logits score; P 32 The values at all positions in the spatial dimension are averaged to obtain a two-dimensional vector X3 containing the image fault logits score and the image fault-free logits score; then, the three-level two-dimensional vectors are added element by element to obtain the comprehensive logits vector X4; finally, the comprehensive logits vector is converted into a probability distribution through the Softmax activation function to obtain a two-dimensional vector P containing the image prediction fault probability and the image prediction fault-free probability. out , then according to P out Get the image prediction failure probability And the image prediction failure-free probability X4=X1+X2+X3 P out =Softmax(X4) In the above formula, Softmax represents the Softmax activation function; + represents the element-by-element addition operation; The specific steps of S332 are as follows: S3321: Input the power equipment detection feature map V1. First, use the channel shuffling method to perform cross-mixing operations on the channel information of V1 to obtain the channel feature enhancement feature map V2. Then, use the split GLU to perform gate processing on V2 to obtain the channel feature enhancement feature map V3. The specific steps are: first, divide V2 into two parts along the channel dimension V 2_1 and V 2_2 , and then V 2_2 Through the Sigmoid activation function, the gate signal G is obtained, and finally V 2_1 Perform element-wise multiplication with G to obtain V3; then, use a convolution kernel of size 1×1 to perform a convolution operation on V3 to obtain a channel feature enhanced feature map V4; S3322: Perform four branch calculations on V4. In the first branch, first use global average pooling to perform a pooling operation on V4 to obtain a channel feature enhanced feature map V5; then use a convolution kernel of size 1×1 to perform a convolution operation on V5 to obtain a channel feature enhanced feature map V6; then V6 is passed through a Sigmoid activation function to obtain a channel feature enhanced feature map V7; in the second branch, use a convolution kernel of size 3×3 to perform a convolution operation on V4 to obtain a channel feature enhanced feature map V8; then V8 is passed through a Tanh activation function to obtain a channel feature enhanced feature map V9; in the third branch, no operation is performed on V4; then, V7, V9 and V4 are element-wise multiplied to obtain a channel feature enhanced feature map V 11 ; S3323: In the fourth branch, V4 is extracted through the Bottleneck module to obtain the channel feature enhancement feature map V 10 ; Finally, V 11 and V 10 Perform channel-by-channel maximum fusion to obtain and output the channel feature enhancement feature map V 12 , the specific operation is: for V 11 and V 10 For each corresponding channel, traverse all spatial positions on the channel feature map and calculate V 11 and V 10 The maximum value of the element at that position is used as V 12 Elements in The specific steps of S334 are as follows: S3341: Input the power equipment detection feature map Y1. First, convolve Y1 with a 3×3 convolution kernel, a 5×5 convolution kernel, and a 7×7 convolution kernel. Then, concatenate the three convolution results in the channel dimension to obtain the attention anomaly detection feature map Y2. Then, convolve Y2 with a 1×1 convolution kernel to obtain the attention anomaly detection feature map Y3. Then, first process Y3 with the ReLU activation function, and then process the processed result with the Hard Sigmoid activation function to obtain the attention anomaly detection feature map Y4. Y2=Concat(Conv3(Y1),Conv5(Y1),Conv7(Y1)) Y4=HardS(ReLU(Conv1(Y2))) In the above formula, Conv3() represents the convolution operation of the convolution kernel of size 3×3; Conv5() represents the convolution operation of the convolution kernel of size 5×5; Conv7() represents the convolution operation of the convolution kernel of size 7×7; Concat() represents the splicing operation in the channel dimension; Conv1() represents the convolution operation of the convolution kernel of size 1×1; ReLU represents the ReLU activation function; HardS represents the Hard Sigmoid activation function; S3342: Perform element-wise multiplication on Y4 and Y1 to obtain an attention anomaly detection feature map Y5; then perform a depthwise separable convolution operation on Y5 using a 3×3 convolution kernel to obtain an attention anomaly detection feature map Y6; then pass Y6 through a sigmoid activation function to obtain an attention anomaly detection feature map Y7; then perform element-wise multiplication on Y7 and Y5 to obtain an attention anomaly detection feature map Y8; Y8=(Y4⊙Y1)⊙S(DSC3(Y5)) In the above formula, DSC3() represents the depth-wise separable convolution operation of the convolution kernel of size 3×3; S represents the Sigmoid activation function; ⊙ represents the element-wise multiplication operation; S3343: Use the fully connected layer to perform feature mapping on Y8 to obtain the attention anomaly detection feature map Y9; then, Y9 is processed by the Sigmoid activation function to obtain the attention anomaly detection feature map Y 10 ; Finally, Y 10 Perform element-wise multiplication with Y8 to obtain the attention anomaly detection feature map Y 11 ; Y 11 =S(FC(Y8))⊙Y8 In the above formula, S represents the Sigmoid activation function; FC() represents the fully connected layer feature mapping operation; ⊙ represents the element-by-element multiplication operation.

8. The method for detecting abnormalities in power equipment based on multimodal deep learning according to claim 4, characterized in that: The specific steps of S34 are as follows: Input sub-model M1 predicts the fault probability of the audio frequency of the power equipment and audio prediction failure-free probability Sub-model M2 predicts the probability of failure of power equipment vibration and vibration prediction failure-free probability Sub-model M3 predicts the probability of failure of power equipment images And the image prediction failure-free probability First, calculate the mutual information I(M1,M2) between M1 and M2, calculate the mutual information I(M1,M3) between M1 and M3, and calculate the mutual information I(M2,M3) between M2 and M3. Then, adjust the weight of each sub-model according to the mutual information calculation results to obtain the adjusted M1 model weight w adjust (1) M2 model weight w adjust (2) and M3 model weight w adjust (3); Then apply the weighted voting method to calculate the weighted voting probability P of the fault vote (1) and the weighted voting probability P of no fault vote (2); Then the soft voting method is applied to calculate the soft voting probability P of faulty soft (1) and the probability of fault-free soft voting P soft (2); Finally, the results of weighted fusion weighted voting method and soft voting method are weighted to obtain the final prediction result: the overall predicted fault probability P mmwf (1) and the overall predicted failure-free probability P mmwf (2); P mmwf (y)=α*P vote (y)+(1-α)*P soft (y) In the above formula, I(M i ,M j ) indicates the calculation of M i and M j The mutual information between them; y represents the binary classification value of whether there is a fault, where y = 1 means there is a fault and y = 2 means there is no fault; Represents the sub-model M i The predicted probability of category y; P(y) is the prior probability of category y, P(y=1) is equal to the ratio of the number of faulty samples to the total number of samples in the data set, and P(y=2) is equal to the ratio of the number of non-faulty samples to the total number of samples in the data set; w i Represents each sub-model M i The learnable weights; k represents the number of sub-models; w adjust (i) represents the sub-model M after adjustment based on mutual information i Normalized weight of 1(y=y pred,i ) represents the sub-model M i An indicator function indicating whether the prediction is for category y; α represents a learnable parameter used to balance the influence of weighted voting and soft voting.

9. The method for detecting abnormalities in power equipment based on multimodal deep learning according to claim 1, characterized in that: The specific steps of S4 are as follows: S41: Divide the pre-processed multimodal dataset of power equipment in S2 into a training set, a validation set, and a test set according to a certain ratio, ensuring that the samples in the training set, validation set, and test set are independent and non-overlapping; during the division process, ensure that the category distribution of each subset is consistent with the overall dataset to avoid the negative impact of data imbalance on model training; S42: Use a dynamically adjusted hyperparameter scheme: the basic training cycle is set to a certain number of rounds, and a progressive batch expansion strategy is implemented, that is, the initial batch size is set to a small value, and the batch size is gradually increased after each training cycle. The learning rate scheduling uses the cyclic cosine annealing algorithm, starting from a low baseline value and setting the cycle length according to training requirements. This allows the learning rate to decrease smoothly during training, effectively improving model convergence. The optimizer uses NAdam and sets gradient normalization constraints. This prevents excessive gradients by limiting the L2 norm of parameter updates, ensuring the stability of the training process. S43: During the training phase, the input samples in each batch are a five-tuple, consisting of: normalized audio data from the one-dimensional sound sub-dataset, mel-spectrograms from the two-dimensional sound sub-dataset, normalized vibration signals from the one-dimensional vibration sub-dataset, Gram angular field images from the two-dimensional vibration sub-dataset, and images of power equipment from the image sub-dataset. Using these five-tuple samples, the model can learn the correlations between different modalities. During feature extraction, a cross-modal contrast loss function is used to optimize the model, effectively aligning data from different modalities in the feature space and improving multimodal learning capabilities. S44: During the validation phase, a multi-dimensional evaluation protocol is used, with full validation performed every several batches. The validation process includes several test modes: standard data inference, noise interference testing, and modal loss testing. An adaptive early stopping mechanism is also implemented: when the coefficient of variation of the main evaluation metric remains below a preset value for several consecutive validation runs, training is automatically terminated to ensure model convergence and stability. S45: During the testing phase, the model will undergo several rigorous tests on the test set, including: cross-device generalization evaluation, real-time stress testing, model stability and robustness, and performance evaluation under data noise interference; through testing, it is ensured that the model can maintain high reliability and accuracy in actual applications and meet the actual needs of power equipment anomaly detection.

10. The method for detecting abnormalities in power equipment based on multimodal deep learning according to claim 1, characterized in that: The specific steps of S5 are as follows: S51: Input the collected audio data, vibration signals, and images of the power equipment into a power equipment anomaly detection model to obtain an overall predicted probability of failure and an overall predicted probability of no failure; S52: Then a judgment is made. If the overall predicted probability of failure is greater than the overall predicted probability of no failure, the output device is faulty; otherwise, the output device is not faulty.

Images