Transformer anomaly detection method based on multi-modal deep learning

Abstract

The invention belongs to the technical field of transformer anomaly detection, and particularly relates to a transformer anomaly detection method based on multi-modal deep learning, which comprises the following specific steps: acquiring transformer multi-modal data, and preprocessing the transformer multi-modal data to obtain a transformer multi-modal data set D; a transformer anomaly detection model is constructed, the transformer multi-modal data is input into the transformer anomaly detection model, and the transformer anomaly detection model is used for detecting whether the transformer is in a normal state or an abnormal state; training a transformer anomaly detection model based on the constructed transformer multi-modal data set to obtain a trained transformer anomaly detection model; and after the transformer anomaly detection model is trained, detecting whether the transformer is in a normal state or an abnormal state by using the transformer anomaly detection model. According to the invention, a transformer anomaly detection model is used, the operation state of the transformer is sensed comprehensively from multiple dimensions, and false alarm and missing alarm conditions are reduced; and meanwhile, errors of manual inspection are avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of transformer anomaly detection, and in particular to a transformer anomaly detection method based on multimodal deep learning. Background Art

[0002] Transformers are crucial in power systems. They can increase or decrease voltage to meet different power demands, improve power transmission efficiency, and reduce transmission line losses. By increasing the voltage, the same power can be transmitted at a lower current, reducing resistance losses. At the same time, transformers provide electrical isolation to ensure the safety of equipment and personnel, and can also stabilize voltage and reduce the impact of voltage fluctuations on power systems and electrical equipment.

[0003] As the "transfer station" of the power transmission and distribution system, the stable operation of the transformer is of great significance. Therefore, early detection of potential hidden dangers and faults in its operation and reasonable formulation of maintenance strategies based on the actual operating status can effectively prevent faults, extend the life of equipment, and minimize the economic risks caused by transformer aging or failure. However, the traditional and widely used method of manual inspection of transformers has many disadvantages. First, it is inefficient and consumes a lot of time and manpower. Second, there are safety risks. Inspectors work in a high-voltage environment and are prone to electric shock and equipment operation hazards. Third, the accuracy is limited by personal experience and skills, and missed inspections or misjudgments may occur due to human factors.

[0004] In addition, transformer anomaly detection methods based on a single dimension have obvious shortcomings. Such methods cannot fully evaluate the health of the transformer and are prone to misjudgment due to environmental fluctuations or temporary changes. Moreover, single-dimensional detection may lead to false alarms and make it difficult to respond to faults in a comprehensive and objective manner.

[0005] In summary, traditional anomaly detection methods have the following disadvantages:

[0006] 1. The traditional manual inspection method is costly and greatly affected by the environment;

[0007] 2. The single-modal deep learning method cannot fully assess the health of the transformer and is prone to misjudgment due to environmental fluctuations or temporary changes;

[0008] Therefore, a transformer anomaly detection method based on multimodal deep learning is invented, which aims to achieve comprehensive perception and anomaly detection of the transformer operating status by fusing multiple sensor data (such as temperature, sound, oil and gas content, etc.) and using deep learning technology to efficiently process and analyze these multimodal data, where the transformer operation is divided into normal state and abnormal state. Summary of the invention

[0009] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions:

[0010] A transformer anomaly detection method based on multimodal deep learning includes the following specific steps:

[0011] S1: Collect transformer multimodal data and perform preprocessing to obtain transformer multimodal data set D;

[0012] S2: Build a transformer anomaly detection model, input transformer multimodal data into the transformer anomaly detection model, and the transformer anomaly detection model is used to detect whether the transformer is in a normal state or an abnormal state;

[0013] S3: training the transformer anomaly detection model based on the constructed transformer multimodal dataset to obtain a trained transformer anomaly detection model;

[0014] S4: After the transformer anomaly detection model is trained, the transformer anomaly detection model is applied to detect whether the transformer is in a normal state or an abnormal state.

[0015] As a preferred solution of the transformer anomaly detection method based on multimodal deep learning described in the present invention, the specific process of S1 is as follows:

[0016] S11: Collect multi-modal data of transformer;

[0017] S12: Preprocess each of the above-collected speech data, thermal imaging images and concentration values ​​of various gases to obtain Mel spectrum x, thermal imaging image data y and gas content data z, respectively, and form a triplet<x,y,z> ;

[0018] S13: Repeat S11 and S12, and get a triple after each execution.<x,y,z> , all triplets form a transformer multimodal dataset D.

[0019] As a preferred solution of the transformer anomaly detection method based on multimodal deep learning described in the present invention, the specific process of S11 is as follows:

[0020] S111: Using a microphone in the substation to record and collect voice clips generated by the vibration of the transformer during operation, the duration of each voice clip is equal, and transformer voice data is obtained;

[0021] S112: During the collection of voice data from the transformer, a thermal imaging camera is used to take a picture to obtain a thermal imaging picture of the transformer;

[0022] S113: Using a gas sensor, obtaining the concentration value of each gas dissolved in the transformer oil during the period of collecting voice data, wherein the gas dissolved in the transformer oil includes six gases, namely H2, CH4, C2H6, C2H4, C2H2, and C0.

[0023] As a preferred solution of the transformer anomaly detection method based on multimodal deep learning described in the present invention, the specific process of S12 is as follows:

[0024] S121: Perform a series of processing such as pre-emphasis, framing, windowing, fast Fourier transform, and Mel filter filtering on the speech data to obtain a Mel spectrum graph x;

[0025] S122: performing noise reduction processing on the thermal imaging image, so that the thermal imaging image can remove salt and pepper noise and random noise while retaining edge features, thereby obtaining thermal imaging image data y;

[0026] S123: First, the units of the concentration values ​​of various gases are unified, and then the total concentration O of the six gases is calculated.

[0027]

[0028] Among them, B i represents the concentration value of the i-th gas, that is, B1 represents the concentration value of H2, B2 represents the concentration value of CH4, B3 represents the concentration value of C2H6, B4 represents the concentration value of C2H4, B5 represents the concentration value of C2H2, and B6 represents the concentration value of CO;

[0029] Then, calculate the percentage of each gas in the total concentration. The specific calculation formula is as follows:

[0030]

[0031] Among them, P u represents the percentage of the u-th gas in O, that is, P1 represents the percentage of H2 in O, P2 represents the percentage of CH4 in O, P3 represents the percentage of C2H6 in O, P4 represents the percentage of C2H4 in O, P5 represents the percentage of C2H2 in O, and P6 represents the percentage of CO in O;

[0032] Finally, P1, P2, P3, P4, P5 and P6 constitute the gas content data z = [P1, P2, P3, P4, P5, P6];

[0033] S124: The Mel spectrum x, thermal imaging image data y and gas content data z during transformer operation are combined to form a triplet<x,y,z> .

[0034] As a preferred solution of the transformer anomaly detection method based on multimodal deep learning described in the present invention, the specific process of S2 is as follows:

[0035] S21: construct a Mel-spectrogram feature extraction module to obtain a Mel-spectrogram feature vector K1;

[0036] S22: construct a thermal imaging picture data feature extraction module to obtain a thermal imaging picture data feature vector K2;

[0037] S23: Use the Transformer network to fuse K1, K2 and z to obtain the feature vector R;

[0038] S24: Use the status judgment module to determine whether the transformer is in an abnormal state.

[0039] As a preferred solution of the transformer abnormality detection method based on multimodal deep learning described in the present invention, the specific process of S21 is as follows:

[0040] S211: Input M1 to the first GCSM submodule to obtain the Mel frequency spectrum feature map Q 11 ;

[0041] S212: Q 11 Input into the first AMA submodule to obtain the Mel spectrum feature map M 11 ;

[0042] S213: M 11 After interleaving through n-1 GCSM submodules and n-1 AMA submodules, the transformer Mel spectrum feature map M is obtained. 1n ;

[0043] S214: M 11 , M 12 , ..., M 1n Perform a Concat operation along the channel dimension to obtain the transformer Mel spectrum feature map M2; where the number of channels of M2 is nC, the frequency length of M2 is F, and the time length of M2 is T;

[0044] S215: Input M2 into MobileNetV1 to obtain the Mel-spectrogram feature vector K1;

[0045] The execution process of the GCSM submodule in S211 is as follows:

[0046] S2111: For the input M1, where the number of channels of M1 is C, the frequency length of M1 is F, and the time length of M1 is T, firstly, the channels of M1 are grouped by a hyperparameter j to obtain the feature map sequence G = {G1, G2, ..., Gj}, for any feature map G k ∈G,G k The number of channels is G k The frequency length is F, G k The time length is T;

[0047] S2112: For any G k ∈G, first perform a global channel average pooling operation to obtain a matrix g with F rows and T columns k , then g k Input into the Sigmoid function for activation, and get the matrix σ with F rows and T columns k , where σ k is a dynamic weight parameter matrix, σ k Each element in is a weight coefficient automatically generated based on the input Mel spectrum feature map;

[0048] S2113: For σ k With G k Perform Conv1D operation to obtain feature map G k1 , where G k1 The number of channels is l is the channel enhancement multiple, G k1 The frequency length is still F, G k1 The length of time T;

[0049] Among them, G k1 The calculation formula is as follows:

[0050]

[0051] Among them, · represents the dot multiplication operation, Conv1D represents the convolution operation, the first parameter is the feature map, and the second parameter is the number of channels of the output feature map;

[0052] S2114: For G k1 Perform Dwconv3D operation to extract independent spatial features of a single channel and obtain the feature map G k2 , where G k2 The number of channels is G k2 The frequency length is F, G k2 The time length is T;

[0053] S2115: For G k2 Perform Conv1D operation to reduce G k2 The number of channels is obtained, and the feature map G is obtained. k3 , where G k3 The number of channels is Gk3 The frequency length is F, G k3 The time length is T;

[0054] Among them, G k3 The calculation formula is as follows:

[0055]

[0056] S2116: Finally, perform Channel Shuffle operation to obtain Mel spectrum feature map Q 11 , where Q 11 The number of channels is C, Q 11 The frequency length is F, Q 11 The time length is T;

[0057] The execution process of the AMA submodule in S212 is as follows:

[0058] S2121: For input Q 11 , using the first SENet network, Q 11 The channel dimension of Q is enhanced to obtain the channel attention enhanced feature map C1; the second SENet network is used to 11 The frequency dimension of Q is enhanced to obtain the frequency attention enhancement feature map F1; the third SENet network is used to 11 The time dimension of the attention is enhanced to obtain the time attention enhancement feature map T1;

[0059] S2122: performing a Concat operation on C1, F1 and T1 to merge C1, F1 and T1 along the channel dimension, thereby obtaining a Mel frequency spectrum feature map A1, wherein the number of channels of A1 is 3C, the frequency length of A1 is F, and the time length of A1 is T;

[0060] S2123: Perform a convolution operation on A1 through a convolution layer to integrate the information of channel, frequency and time dimensions to obtain the Mel spectrum feature map M 11 , M 11 The number of channels is C, M 11 The frequency length is F, M 11 The time length is T;

[0061] The specific process of S213 is as follows:

[0062] S2131: M 11 Input to the second GCSM submodule to obtain the Mel spectrum feature map Q 12 ;

[0063] S2132: Q 12Input to the second AMA submodule to obtain the Mel spectrum feature map M 12 ;

[0064] S2133: M 12 Input to the third GCSM submodule to obtain the Mel spectrum feature map Q 13 ;

[0065] S2134: Q 13 Input to the third AMA submodule to obtain the Mel spectrum feature map M 13 ; Until the Mel spectrum feature map Q is obtained by the nth GCSM submodule 1n ;

[0066] S2135: Q 1n Input to the nth AMA submodule to obtain the transformer Mel spectrum feature map M 1n ;

[0067] The specific process of S215 is as follows:

[0068] S2151: In MobileNetV1, the basic spatial features are first extracted through 3x3 convolution, and the feature map size is reduced to obtain the feature map M 21 ;

[0069] S2152: Through the deep convolution layer, M 21 Each channel of M is convolved independently to extract the spatial features of each channel and obtain the feature map M 22 ;

[0070] S2153: M is transformed through 1x1 convolution 22 Integrate the features of different channels to capture cross-channel dependencies so as to extract key spatial and channel information from M2;

[0071] S2154: After multiple layers of convolution operations, the extracted features are aggregated into a feature vector K1 through a fully connected layer. K1 contains the key information extracted from M2, where the extracted key information includes the frequency distribution information, energy distribution, time variation characteristics and spatial relationship of M2.

[0072] As a preferred solution of the transformer anomaly detection method based on multimodal deep learning described in the present invention, the specific process of S22 is as follows:

[0073] S221: Input Y1 into the OCA submodule to obtain the thermal imaging feature map Y2;

[0074] S222: Input Y2 into the first GCSM submodule to obtain a thermal imaging feature map Y3, and then input Y3 into the second GCSM submodule to obtain a thermal imaging feature map Y4, until passing through the wth GCSM submodule to obtain a thermal imaging feature map Y w+2 , where Y w+2 The number of channels is U, Y w+2 The height is H, Y w+2 The width is W;

[0075] S223: Y w+2 Input into MobileNetV1 to obtain the thermal imaging image data feature vector K2;

[0076] The execution process of the OCA submodule in S221 is as follows:

[0077] S2211: For the input thermal imaging feature map Y1, where the number of channels of Y1 is U, the height of Y1 is H, and the width of Y1 is W; for each channel of Y1, perform global maximum pooling and global average pooling respectively to extract global features in the spatial dimension, and finally obtain two vectors: maximum pooling result y max And the average pooling result y avg , where y max is a vector with length U, y avg is a vector with a length of U;

[0078] S2212: For y max Normalize and get a vector max of length U; max Normalize and get a vector avg of length U;

[0079] S2213: Input max and avg into the activation function Tswish to obtain an attention vector α of length U c :

[0080]

[0081] Where, · is the dot multiplication operation; is a hyperparameter with a value range of (0, 1]. The advantage of using the Swish activation function is that it avoids discontinuity near the zero point, so that the thermal imaging image data can better retain the characteristics. max and w avg is a learnable weight parameter that can be set by the initial weight w max >w avg, giving max a larger weight, making the weight of the global maximum pooling contribute more significantly to the OCA submodule; allowing the OCA submodule to pay attention to the most significant features in each channel, while also paying attention to the global statistical information of each channel;

[0082] S2214: α c Multiply element-by-element with Y1 to obtain the thermal imaging feature map Y2, where the number of channels of Y2 is U, the height of Y2 is H, and the width of Y2 is W;

[0083] Y2=Y1⊙α c

[0084] Among them, ⊙ represents the element-by-element multiplication operation, that is, the broadcast mechanism converts α c Expanded to the dimension of y, finally, the thermal imaging feature map Y2 enhanced by the attention mechanism is obtained.

[0085] As a preferred solution of the transformer abnormality detection method based on multimodal deep learning described in the present invention, the specific process of S23 is as follows:

[0086] S231: Input K1, K2 and z to the Transformer network. First, feature encode K1, K2 and z in the encoding layer of the Transformer network to obtain the feature vector K 11 , K 22 with z1;

[0087] S232: K 11 , K 22 and z1 are input into the self-attention mechanism module, K 11 , K 22 and z1 will generate their own query, key, and value vectors through linear transformation; 11 , get the query vector q1, key vector k1 and value vector v1; for K 22 , we get the query vector q2, key vector k2 and value vector v2; for z1, we get the query vector q3, key vector k3 and value vector v3;

[0088] S233: Based on K 11 , K 22 Calculate K with the query, key and value vectors of z1 11 , K 22 The correlation with z1 gives K 11 , K 22 and the weighted value vector of z1, representing v 11 , v 22 , v 33 , the calculation formula is as follows:

[0089]

[0090]

[0091] in, is the transpose of the s-th key vector, is the dimension of the sth key vector;

[0092] S234: V 11 、v 22 and v 33 Input to the decoding layer of the Transformer network for feature decoding, v 11 After decoding, we get v d11 ; for v 22 After decoding, we get v d22 , v 33 After decoding, we get v d33 ;

[0093] S235: V d11 、v d22 and v d33 Perform concatenation operation to obtain a feature vector r = [v d11 , v d22 , v d33 ], r is a feature representation that integrates all modal information;

[0094] S236: r is input into a feedforward neural network for processing to obtain a feature vector R, wherein the feedforward neural network is composed of several fully connected layers and activation function ReLU, which further optimizes the representation of the features; R integrates the key information of K1, K2 and z, wherein the key information includes the association information between speech and temperature, the association information between temperature and gas, and the association information between gas and speech.

[0095] As a preferred solution of the transformer abnormality detection method based on multimodal deep learning described in the present invention, the specific process of S24 is as follows:

[0096] S241: Using a state judgment module to detect the state of the transformer during operation, wherein the state judgment module is a fully connected neural network;

[0097] S242: Input R to FNN, perform nonlinear mapping on the features through the hidden layer to extract features; use multiple hidden layers to extract higher-level features; the output layer receives the output of the hidden layer, performs a full connection operation and uses the Sigmoid activation function to determine whether the transformer is in an abnormal state.

[0098] As a preferred solution of the transformer anomaly detection method based on multimodal deep learning described in the present invention, the specific process of S3 is as follows:

[0099] S31: In the training process of the transformer abnormal state detection model, first initialize the parameters of all models and set the relevant hyperparameters, including n in the Mel spectrum feature extraction module, w in the thermal imaging image data feature extraction module, training rounds, batch size, optimizer selection, and learning rate;

[0100] S32: Divide the transformer multimodal data set into a training set, a validation set, and a test set according to a certain ratio to ensure that the three do not overlap with each other; then, divide the training set and the validation set data into multiple batches, input a batch of training data into the model for training each time, and calculate the training loss value of the batch;

[0101] S33: After all batches of the entire training set have completed a round of training, the validation set data is input into the model in batches to obtain the corresponding validation set loss value; the validation set loss value is used to monitor whether the model is overfitting and adjust the training strategy as needed;

[0102] S34: Compare the training results of the transformer anomaly detection model under the same training rounds, batch size, optimizer selection, and learning rate, and finally determine the values ​​of n and w.

[0103] Compared with existing technologies:

[0104] 1. Using the transformer anomaly detection model, by integrating multiple sensor data (such as voice, temperature, oil and gas content), it can fully perceive the operating status of the transformer from multiple dimensions, reduce false alarms and missed alarms, and avoid problems caused by lack of experience or human errors in manual inspections;

[0105] 2. The fusion of multimodal data enhances the model's adaptability to environmental fluctuations and temporary changes, and enables accurate judgment of the transformer status, thereby ensuring the safe and stable operation of the power system, extending equipment life, and reducing economic losses caused by faults. BRIEF DESCRIPTION OF THE DRAWINGS

[0106] Figure 1 It is a schematic diagram of the process of the present invention;

[0107] Figure 2 This is a schematic diagram of a transformer abnormal state detection model according to the present invention;

[0108] Figure 3 This is a schematic diagram of a Mel-spectrogram feature extraction module of the present invention;

[0109] Figure 4 This is a schematic diagram of the GCSM submodule of the present invention;

[0110] Figure 5 This is a schematic diagram of the AMA submodule of the present invention;

[0111] Figure 6 This is a schematic diagram of a thermal imaging picture data feature extraction module of the present invention;

[0112] Figure 7 Schematic diagram of the OCA submodule of the present invention. DETAILED DESCRIPTION

[0113] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0114] The present invention provides a transformer anomaly detection method based on multimodal deep learning, please refer to Figure 1-Figure 7 , including the following specific steps:

[0115] S1: Collect transformer multimodal data and perform preprocessing to obtain transformer multimodal data set D; collect voice data, thermal imaging images and concentration values ​​of various gases dissolved in oil generated by transformer vibration in the same time period for transformers in normal and abnormal states, and then preprocess the multimodal raw data to obtain a transformer multimodal data set D, as follows:

[0116] The specific process of S1 is as follows:

[0117] S11: Collect multi-modal data of transformer;

[0118] The specific process of S11 is as follows:

[0119] S111: Using a microphone in the substation to record and collect voice clips generated by the vibration of the transformer during operation, the duration of each voice clip is equal, and transformer voice data is obtained;

[0120] S112: During the collection of voice data from the transformer, a thermal imaging camera is used to take a picture to obtain a thermal imaging picture of the transformer;

[0121] S113: using a gas sensor to obtain a concentration value of each gas dissolved in the transformer oil during the period of collecting voice data, wherein the gas dissolved in the transformer oil includes six gases, namely H2, CH4, C2H6, C2H4, C2H2, and CO;

[0122] Including but not limited to the following embodiments:

[0123] A microphone is installed on the supporting structure or foundation of the transformer, and then the microphone collects voice clips generated by the vibration of the transformer during operation. The duration of each voice clip is 20 seconds. During the collection of voice data, a professional thermal imaging camera is used to take thermal imaging pictures with a picture size of 512x512. The thermal imaging camera is fixed on the front of the transformer, 5m away from the transformer. Finally, a gas sensor is used to check the concentration values ​​of H2, CH4, C2H6, C2H4, C2H2, and CO in the dissolved gas in the transformer oil recorded by the gas sensor during the collection of voice data.

[0124] S12: Preprocess each of the above-collected speech data, thermal imaging images and concentration values ​​of various gases to obtain Mel spectrum x, thermal imaging image data y and gas content data z, respectively, and form a triplet<x,y,z> ;

[0125] The specific process of S12 is as follows:

[0126] S121: Perform a series of processing such as pre-emphasis, framing, windowing, fast Fourier transform, and Mel filter filtering on the speech data to obtain a Mel spectrum graph x;

[0127] Among them: converting speech data into a Mel-spectrogram is a common operation in the field of speech processing and is not an innovative point of the present invention;

[0128] Mel spectrogram: Mel spectrogram is a feature representation that converts speech data into a spectrogram and maps it to the Mel scale. It is widely used in speech processing and speech analysis. The size of the Mel spectrogram is determined by the number of channels, frequency length, and time length. The number of channels is determined by the number of voice channels or by the Mel filter group that further processes or combines information from different frequency bands. It has no unit. The frequency length reflects the number of divided Mel frequency bands, which is determined by the number of Mel filter groups, such as 40 or 64, and has no unit. The time length is the number of discrete frames in the time dimension, and has no unit.

[0129] S122: performing noise reduction processing on the thermal imaging image, so that the thermal imaging image can remove salt and pepper noise and random noise while retaining edge features, thereby obtaining thermal imaging image data y;

[0130] Including but not limited to the following embodiments:

[0131] When using median filtering to reduce noise in thermal imaging images, median filtering slides over the image pixel by pixel through a fixed-size window (such as 3×3), extracts all pixel values ​​in the window each time, sorts them by size, takes the middle value as the median, and then uses the median to replace the original value of the center pixel of the window; median filtering can effectively remove salt and pepper noise and random noise while retaining edge features;

[0132] S123: First, the units of the concentration values ​​of various gases are unified, and then the total concentration O of the six gases is calculated.

[0133]

[0134] Among them, B i represents the concentration value of the i-th gas, that is, B1 represents the concentration value of H2, B2 represents the concentration value of CH4, B3 represents the concentration value of C2H6, B4 represents the concentration value of C2H4, B5 represents the concentration value of C2H2, and B6 represents the concentration value of CO;

[0135] Then, calculate the percentage of each gas in the total concentration. The specific calculation formula is as follows:

[0136]

[0137] Among them, P u represents the percentage of the u-th gas in O, that is, P1 represents the percentage of H2 in O, P2 represents the percentage of CH4 in O, P3 represents the percentage of C2H6 in O, P4 represents the percentage of C2H4 in O, P5 represents the percentage of C2H2 in O, and P6 represents the percentage of CO in O;

[0138] Finally, P1, P2, P3, P4, P5 and P6 constitute the gas content data z = [P1, P2, P3, P4, P5, P6];

[0139] Including but not limited to the following embodiments:

[0140] After unifying the units of various gas concentration values, the concentration values ​​of H2 are 120ppm, CH4 are 80ppm, C2H6 are 45ppm, C2H4 are 30ppm, C2H2 are 1ppm, and CO are 300ppm; adding these gas concentrations, the total concentration O is 576ppm; ppm (parts per million) is a unit for expressing the concentration of a substance, usually used to describe the proportion of a certain component of a gas to the total amount; it indicates the number of units of a substance contained in every million units;

[0141] Calculate the percentage of each gas in the total concentration and get the percentage of H2 in O to be 0.83, the percentage of CH4 in O to be 13.89, the percentage of C2H6 in O to be 7.81, the percentage of C2H4 in O to be 5.21, the percentage of C2H2 in O to be 0.17, and the percentage of CO in O to be 52.08;

[0142] Finally, we get the gas content data z = [0.83, 13.89, 7.81, 5.21, 0.17, 52.08];

[0143] S124: The Mel spectrum x, thermal imaging image data y and gas content data z during transformer operation are combined to form a triplet<x,y,z> ;

[0144] S13: Repeat S11 and S12, and get a triple after each execution.<x,y,z> , all triples form a transformer multimodal dataset D;

[0145] S2: Construct a transformer anomaly detection model, input the transformer multimodal data into the transformer anomaly detection model, and the transformer anomaly detection model is used to detect whether the transformer is in a normal state or an abnormal state; the transformer anomaly detection model is as follows: Figure 2 As shown; for any triple<x,y,z> ∈D, use the Mel spectrum feature extraction module to extract features from x and obtain the Mel spectrum feature vector K1; use the thermal imaging image data feature extraction module to extract features from y and obtain the thermal imaging image data feature vector K2; then use the Transformer network to fuse K1, K2 and z to obtain the feature vector R; finally, use the state judgment module to detect whether the transformer is in a normal state or an abnormal state;

[0146] The specific process of S2 is as follows:

[0147] S21: construct a Mel-spectrogram feature extraction module to obtain a Mel-spectrogram feature vector K1;

[0148] First, the triple<x,y,z> The x in the figure is used as the Mel spectrum feature map M1, and M1 is input into the first GCSM submodule to obtain the Mel spectrum feature map Q 11 , Q 11 After inputting into the first AMA submodule, the Mel spectrum feature map M is obtained. 11 , M 11 After inputting into the second GCSM submodule, the Mel spectrum feature map Q is obtained. 12 , Q 12 After inputting into the second AMA submodule, the Mel spectrum feature map M is obtained. 12 , until M is obtained after the nth AMA submodule 1n ; Then all outputs M that pass through the AMA submodule 11 、M 12 …M 1n Perform the Concat operation to obtain the Mel spectrum feature map M2, and then input M2 into MobileNetV1 to obtain the Mel spectrum map feature vector K1.

[0149] The Mel-spectrogram feature extraction module is first composed of n GCSM submodules and n AMA submodules interlaced in series, and then combined with Concat and MobileNetV1, where n is determined during the training process of the transformer anomaly detection model. The structure is as follows Figure 3 As shown;

[0150] Among them, Concat is a feature fusion operation commonly used in deep learning models. It is widely used for its efficient structural design and low computational complexity. By splicing multiple feature maps in the channel dimension, information from different layers or paths is integrated together, enhancing the expressiveness of the model. Concat splices multiple convolutional feature maps in the channel dimension, which is widely used in multi-path feature fusion and cross-layer feature connection, and can enhance the expressiveness of the model.

[0151] MobileNetV1 is a widely used lightweight deep learning model, known for its efficient architecture design and low computational complexity, which can significantly reduce model parameters and computational complexity. Thanks to this design, MobileNetV1 can provide strong performance with low computational overhead on resource-constrained devices (such as mobile terminals), becoming a model of lightweight model design;

[0152] Including but not limited to the following embodiments:

[0153] Confirmation of n: During the training process, the value of n can be selected as one of {4, 6, 8, 16}, and then the optimal value of n is determined based on the comparison of the judgment effect of the model on the transformer state. In this embodiment, n=8 is used first;

[0154] Mel spectrum graph M1, the number of channels of M1 is 64, the frequency length is 40, and the time length is 2000; First, M1 is input into the first GCSM submodule to obtain the Mel spectrum graph Q 11 , its size and channel remain unchanged; then, Q 11 Input to the first AMA submodule to obtain the Mel spectrum map M 11 , and then continue through the second GCSM submodule and the second AMA submodule to obtain the Mel spectrum Q 12 and M 12 , until the 8th AMA submodule, we get M 18 ; All Mel-spectrograms M processed by the AMA submodule 11 , M 12 , ..., M 18After concatenation, a new Mel-spectrogram M2 is obtained, where the number of channels of M2 is 192, the frequency length of M2 is 40, and the time length of M2 is 2000; finally, M2 is input into MobileNetV1 to finally obtain the Mel-spectrogram feature vector K1, and the vector length of K1 is 1000;

[0155] The specific process of S21 is as follows:

[0156] S211: Input M1 to the first GCSM submodule to obtain the Mel frequency spectrum feature map Q 11 The present invention designs a GCSM (Grouped Convolution and Shuffle Module) submodule; the GCSM submodule is used to extract the features of the Mel spectrum graph to obtain a Mel spectrum feature graph with enhanced features; wherein the structure of the GCSM submodule is as follows Figure 4 As shown;

[0157] The execution process of the GCSM submodule in S211 is as follows:

[0158] S2111: For the input M1, where the number of channels of M1 is C, the frequency length of M1 is F, and the time length of M1 is T, firstly, the channels of M1 are grouped by a hyperparameter j to obtain the feature map sequence G = {G1, G2, ..., G j}, for any feature map G k ∈G,G k The number of channels is G k The frequency length is F, G k The time length is T;

[0159] S2112: For any G k ∈G, first perform a global channel average pooling operation to obtain a matrix g with F rows and T columns k , then g k Input into the Sigmoid function for activation, and get the matrix σ with F rows and T columns k , where σ k is a dynamic weight parameter matrix, σ k Each element in is a weight coefficient automatically generated based on the input Mel spectrum feature map;

[0160] Beneficial effects of S2112: By pooling the global channel average of the channel, the expression ability of the channel features can be enhanced, and the σ k The real-time adjustment of the convolution kernel weights enables the GCSM module to enhance or weaken features according to the characteristics of the input feature map;

[0161] Among them, Global Channel Average Pooling (GCAP) is a pooling operation for the channel dimension, which is used to compress the information in the multi-channel feature map into a two-dimensional space. It can retain the global features of each spatial position; it averages the feature values ​​on all channels of each spatial position and then generates a two-dimensional space;

[0162] S2113: For σ k With G k Perform Conv1D operation to obtain feature map G k1 , where G k1 The number of channels is l is the channel enhancement factor (can be 3), G k1 The frequency length is still F, G k1 The length of time T;

[0163] Among them, G k1 The calculation formula is as follows:

[0164]

[0165] Among them, · represents the dot multiplication operation, Conv1D represents the convolution operation, the first parameter is the feature map, and the second parameter is the number of channels of the output feature map;

[0166] The beneficial effect of S2113: Improve G by increasing the number of channels k The ability to express characteristics;

[0167] S2114: For G k1 Perform Dwconv3D operation to extract independent spatial features of a single channel and obtain the feature map G k2 , where G k2 The number of channels is G k2 The frequency length is F, G k2 The time length is T;

[0168] Beneficial effects of S2114: k1 It extracts spatial features independently for each channel, retains the unique information of each channel, and significantly reduces the amount of calculation and parameters, providing higher computing efficiency and flexibility, which helps to improve the feature extraction effect, thereby further enhancing the performance of the model;

[0169] S2115: For G k2 Perform Conv1D operation to reduce G k2 The number of channels is obtained, and the feature map G is obtained. k3 , where G k3 The number of channels is G k3 The frequency length is F, G k3 The time length is T;

[0170] Beneficial effects of S2115: By reducing the number of channels, key information is screened out and redundant features are discarded, thereby improving the representation ability of features, reducing the risk of overfitting, and enhancing the stability of gradient propagation;

[0171] Among them, G k3 The calculation formula is as follows:

[0172]

[0173] S2116: Finally, perform Channel Shuffle operation to obtain Mel spectrum feature map Q 11 , where Q 11 The number of channels is C, Q 11 The frequency length is F, Q 11 The time length is T;

[0174] Among them, Conv1D: is a one-dimensional convolution operation, whose convolution kernel slides along the time axis or sequence axis of the feature map to extract local features in the sequence data; the convolution operation acts on all input channels at the same time and integrates information across channels by sharing weights; it can efficiently capture local patterns in sequence data while retaining global context information;

[0175] Dwconv3D: is a three-dimensional depth-wise separable convolution operation, whose convolution kernel acts independently on each input channel, extracting independent spatial features from a single channel of the input feature map without involving cross-channel weight sharing; it retains the unique information of each channel of the input feature map while significantly reducing the amount of computation and parameters; it can efficiently capture local patterns of three-dimensional data while reducing computational complexity;

[0176] Channel Shuffle: is a simple and efficient channel shuffling operation used to enhance channel interaction in group convolution. By grouping, rearranging, and restoring channels, Channel Shuffle can significantly alleviate the problem of information isolation in group convolution while maintaining computational efficiency.

[0177] S212: Q 11 Input into the first AMA submodule to obtain the Mel spectrum feature map M 11 The present invention designs an AMA (All Dimensions Attention) submodule; the AMA submodule is used to enhance the features of each dimension of the Mel spectrum graph, and obtain a Mel spectrum feature graph with enhanced features; wherein the structure of the AMA submodule is as follows Figure 5 As shown;

[0178] The execution process of the AMA submodule in S212 is as follows:

[0179] S2121: For input Q 11 , using the first SENet network, Q 11 The channel dimension of Q is enhanced to obtain the channel attention enhanced feature map C1; the second SENet network is used to 11 The frequency dimension of Q is enhanced to obtain the frequency attention enhancement feature map F1; the third SENet network is used to 11 The time dimension of the attention is enhanced to obtain the time attention enhancement feature map T1;

[0180] Beneficial effects of S2121: By using the SENet network, adaptive weight coefficients can be generated for the channel dimension, frequency dimension and time dimension to automatically adjust the weight of each channel, time step and frequency component; these weights reflect the importance of each dimension, thereby enhancing the focus of the Mel-spectrogram feature extraction module on key features;

[0181] S2122: performing a Concat operation on C1, F1 and T1 to merge C1, F1 and T1 along the channel dimension, thereby obtaining a Mel frequency spectrum feature map A1, wherein the number of channels of A1 is 3C, the frequency length of A1 is F, and the time length of A1 is T;

[0182] Beneficial effects of S2122: Through the Concat operation, the feature information of each dimension after SENet can be fully preserved;

[0183] S2123: Perform a convolution operation on A1 through a convolution layer to integrate the information of channel, frequency and time dimensions to obtain the Mel spectrum feature map M 11 , M 11 The number of channels is C, M 11 The frequency length is F, M 11 The time length is T;

[0184] Beneficial effects of S2123: By integrating features of various dimensions through convolutional layers, the Mel-spectrogram feature extraction module is optimized to understand and represent the Mel-spectrogram;

[0185] in, Figure 5 The three SENet networks in are exactly the same;

[0186] SENet (Squeeze-and-Excitation Network) is a network that improves the information interaction and feature expression capabilities between the same dimensions. By introducing the two steps of Squeeze and Excitation, the selected dimensions are adaptively weighted, which significantly improves the feature representation capability. Through these two steps, the network can learn the importance of each dimension, thereby enhancing the expression of key features and suppressing irrelevant features, thereby improving the performance and generalization ability of the network.

[0187] S213: M 11 After interleaving through n-1 GCSM submodules and n-1 AMA submodules, the transformer Mel spectrum feature map M is obtained. 1n ;

[0188] The specific process of S213 is as follows:

[0189] S2131: M 11 Input to the second GCSM submodule to obtain the Mel spectrum feature map Q 12 ;

[0190] S2132: Q 12 Input to the second AMA submodule to obtain the Mel spectrum feature map M 12 ;

[0191] S2133: M 12 Input to the third GCSM submodule to obtain the Mel spectrum feature map Q 13 ;

[0192] S2134: Q 13 Input to the third AMA submodule to obtain the Mel spectrum feature map M 13 ; Until the Mel spectrum feature map Q is obtained by the nth GCSM submodule 1n ;

[0193] S2135: Q 1n Input to the nth AMA submodule to obtain the transformer Mel spectrum feature map M 1n ;

[0194] S214: M 11 ,M 12 ,…,M 1n Perform a Concat operation along the channel dimension to obtain the transformer Mel spectrum feature map M2; where the number of channels of M2 is nC, the frequency length of M2 is F, and the time length of M2 is T;

[0195] S215: Input M2 into MobileNetV1 to obtain the Mel-spectrogram feature vector K1;

[0196] The specific process of S215 is as follows:

[0197] S2151: In MobileNetV1, the basic spatial features are first extracted through 3x3 convolution, and the feature map size is reduced to obtain the feature map M 21 ;

[0198] S2152: Through the deep convolution layer, M 21 Each channel of M is convolved independently to extract the spatial features of each channel and obtain the feature map M 22 ;

[0199] S2153: M is transformed through 1x1 convolution 22 Integrate the features of different channels to capture cross-channel dependencies so as to extract key spatial and channel information from M2;

[0200] S2154: After multiple layers of convolution operations, the extracted features are aggregated into a feature vector K1 through a fully connected layer. K1 contains key information extracted from M2, where the extracted key information includes frequency distribution information, energy distribution, time variation characteristics and spatial relationship of M2;

[0201] S22: construct a thermal imaging picture data feature extraction module to obtain a thermal imaging picture data feature vector K2;

[0202] First, the triple<x,y,z> The y is used as the thermal imaging feature map Y1, and Y1 is input into the OCA submodule to obtain the thermal imaging feature map Y2. Y2 is input into the first GCSM submodule to obtain the thermal imaging feature map Y3. Y3 is input into the second GCSM submodule to obtain the thermal imaging feature map Y4, until the thermal imaging feature map Y is obtained after passing through the wth GCSM submodule. w+2 , and finally Y w+2 Input into MobileNetV1 to obtain the thermal imaging image data feature vector K2;

[0203] The thermal imaging image data feature extraction module is composed of an OCA submodule, w GCSM submodules and MobileNetV1 in series, where w is determined during the training process of the transformer anomaly detection model. The structure is as follows: Figure 6 As shown;

[0204] The specific process of S22 is as follows:

[0205] S221: Input Y1 into the OCA submodule to obtain the thermal imaging feature map Y2; the present invention designs an OCA (Only Channel Attention) submodule; the OCA submodule is used to enhance the channel feature of Y1 to obtain the thermal imaging feature map Y2, wherein the OCA submodule is as follows: Figure 7 As shown;

[0206] The execution process of the OCA submodule in S221 is as follows:

[0207] S2211: For the input thermal imaging feature map Y1, where the number of channels of Y1 is U, the height of Y1 is H, and the width of Y1 is W; for each channel of Y1, perform global maximum pooling and global average pooling respectively to extract global features in the spatial dimension, and finally obtain two vectors: maximum pooling result y max And the average pooling result y avg , where y max is a vector with length U, y avg is a vector with a length of U;

[0208] Beneficial effects of S2211: The most significant features in each channel can be extracted through global maximum pooling, and global average pooling can extract global statistical information of each channel;

[0209] Among them, global maximum pooling is a pooling operation that takes the maximum value of all values ​​in the spatial dimensions (i.e., height H and width W) of the input feature map, thereby compressing the two-dimensional feature map into a one-dimensional channel vector; this operation can extract the most significant features in each channel;

[0210] Global average pooling: is a pooling operation that calculates the average of all values ​​in the spatial dimensions (i.e., height H and width W) of the input feature map, thereby compressing the two-dimensional feature map into a one-dimensional channel vector; this operation can extract global statistical information for each channel;

[0211] S2212: To yma x Normalize and get a vector max of length U; x Normalize and get a vector avg of length U;

[0212] Among them, normalization: is a data preprocessing operation, which adjusts the value of the input data to the range of [0, 1] to eliminate the influence of the numerical scale between different features or samples;

[0213] S2213: Input max and avg into the activation function Tswish to obtain an attention vector α of length U c :

[0214]

[0215] Where, · is the dot multiplication operation; is a hyperparameter with a value range of (0, 1]. The advantage of using the Swish activation function is that it avoids discontinuity near the zero point, so that the thermal imaging image data can better retain the characteristics. max and w avg is a learnable weight parameter that can be set by the initial weight w max >w avg , giving max a larger weight, making the weight of the global maximum pooling contribute more significantly to the OCA submodule; allowing the OCA submodule to pay attention to the most significant features in each channel, while also paying attention to the global statistical information of each channel;

[0216] S2214: α c Perform element-by-element multiplication with Y1 to obtain the thermal imaging feature map Y2, where the number of channels of Y2 is U, the height of Y2 is H, and the width of Y2 is W;

[0217] Y2=Y1⊙α c

[0218] Among them, ⊙ represents the element-by-element multiplication operation, that is, the broadcast mechanism converts α c Expand to the dimension of y, and finally obtain the thermal imaging feature map Y2 enhanced by the attention mechanism;

[0219] Beneficial effects of S2214: Through the OCA submodule, the thermal imaging image data feature extraction module can more effectively focus on important information between channels and improve the quality of feature expression;

[0220] Broadcasting mechanism: refers to a mechanism in which when performing operations on vectors of different dimensions or sizes, the smaller vectors are expanded to match the shape of the larger tensor through certain rules, thereby achieving element-by-element operations;

[0221] S222: Input Y2 into the first GCSM submodule to obtain a thermal imaging feature map Y3, and then input Y3 into the second GCSM submodule to obtain a thermal imaging feature map Y4, until passing through the wth GCSM submodule to obtain a thermal imaging feature map Y w+2 , where Y w+2 The number of channels is U, Y w+2 The height is H, Y w+2 The width is W;

[0222] S223: Y w+2 Input into MobileNetVl to obtain the thermal imaging image data feature vector K2;

[0223] S23: Use the Transformer network to fuse K1, K2 and z to obtain the feature vector R;

[0224] The specific process of S23 is as follows:

[0225] S231: Input K1, K2 and z to the Transformer network. First, feature encode K1, K2 and z in the encoding layer of the Transformer network to obtain the feature vector K 11 , K 22 with z1;

[0226] S232: K 11 , K 22 and z1 are input into the self-attention mechanism module, K 11 , K 22 and z1 will generate their own query, key, and value vectors through linear transformation; 11 , get the query vector q1, key vector k1 and value vector v1; for K 22 , we get the query vector q2, key vector k2 and value vector v2; for z1, we get the query vector q3, key vector k3 and value vector v3;

[0227] S233: Based on K 11 , K 22 Calculate K with the query, key, and value vectors of z1 11 , K 22 The correlation with z1 gives K 11 , K 22 and the weighted value vector of z1, representing v 11 , v 22 , v 33 , the calculation formula is as follows:

[0228]

[0229]

[0230] in, is the transpose of the sth key vector, is the dimension of the sth key vector;

[0231] S234: V 11 、v 22 and v 33 Input to the decoding layer of the Transformer network for feature decoding, v 11 After decoding, we get v d11 ; for v 22 After decoding, we get v d22 , v 33 After decoding, we get v d33 ;

[0232] S235: V d11 、v d22 and v d33 Perform concatenation operation to obtain a feature vector r = [v d11 , v d22 , v d33 ], r is a feature representation that integrates all modal information;

[0233] S236: r is input into a feedforward neural network for processing to obtain a feature vector R, wherein the feedforward neural network is composed of several fully connected layers and an activation function ReLU, which further optimizes the representation of the feature; R integrates the key information of K1, K2 and z, wherein the key information includes the association information between speech and temperature, the association information between temperature and gas, and the association information between gas and speech;

[0234] Among them, the Transformer network is a deep learning architecture widely used in natural language processing (NLP) and other tasks, and is famous for its efficient self-attention mechanism and strong modeling ability. It completely abandons the traditional recurrent neural network (RNN) and convolutional neural network (CNN) structure, and instead uses the self-attention mechanism to model long-term dependencies in sequence data. The advantages of the Transformer network in multimodal data fusion lie in its powerful self-attention mechanism, efficient cross-modal modeling ability, long-distance dependency capture, parallel computing ability, and good scalability and adaptability.

[0235] S24: using the state judgment module to determine whether the transformer is in an abnormal state;

[0236] The specific process of S24 is as follows:

[0237] S241: Use a state judgment module to detect the state of the transformer during operation, wherein the state judgment module is a fully connected neural network (FNN);

[0238] S242: Input R to FNN, perform nonlinear mapping on features through hidden layers, and extract features; use multiple hidden layers to extract higher-level features; the output layer receives the output of the hidden layer, performs full connection operations, and uses Sigmoid activation functions to determine whether the transformer is in an abnormal state;

[0239] Among them, the fully connected neural network (FNN) is a basic model commonly used in deep learning tasks. The neurons in each layer are connected to each neuron in the previous layer, and nonlinear transformation is achieved through weighted summation and activation function. It can automatically learn complex input-output relationships and effectively integrate features of different modalities to form a unified representation. The fully connected network has strong adaptability and can process multimodal data. It can optimize parameters through back propagation and is widely used in tasks such as feature fusion and classification. In binary classification tasks, the fully connected network can effectively perform binary judgment through the Sigmoid activation function of the output layer, which is suitable for processing binary classification problems with clear labels.

[0240] S3: training the transformer anomaly detection model based on the constructed transformer multimodal dataset to obtain a trained transformer anomaly detection model;

[0241] The specific process of S3 is as follows:

[0242] S31: In the training process of the transformer abnormal state detection model, first initialize the parameters of all models and set the relevant hyperparameters, including n in the Mel spectrum feature extraction module, w in the thermal imaging image data feature extraction module, training rounds (epoch), batch size (batch size), optimizer selection, learning rate, etc.;

[0243] S32: Divide the transformer multimodal data set into a training set, a validation set, and a test set according to a certain ratio to ensure that the three do not overlap with each other; then, divide the training set and the validation set data into multiple batches, input a batch of training data into the model for training each time, and calculate the training loss value (loss) of the batch;

[0244] S33: After all batches of the entire training set have completed a round of training, the validation set data is input into the model in batches to obtain the corresponding validation set loss value (batch_loss); the validation set loss value is used to monitor whether the model is overfitting and adjust the training strategy as needed, such as terminating training early or adjusting the learning rate;

[0245] S34: Compare the training results of the transformer anomaly detection model under the same training rounds, batch size, optimizer selection, and learning rate, and finally determine the values ​​of n and w;

[0246] Including but not limited to the following embodiments:

[0247] The transformer multimodal dataset is divided into training set, validation set and test set according to a certain ratio to ensure that the three do not overlap with each other, where the training set accounts for 70% of the total dataset, and the validation set and test set each account for 15%; the training round (epoch) is set to 150, and the batch size (batch) is set to 8; the optimizer uses stochastic gradient descent (SGD), the initial learning rate is set to 0.01, and the weight decay is set to 0.0005; if the validation set loss is not improved for 5 consecutive rounds, the learning rate is halved; when the validation set loss no longer decreases, it is considered that the model has basically converged and the training is completed; comparing the training effect of the transformer anomaly detection model, the value of n is 8 and the value of w is 4;

[0248] S4: After the transformer anomaly detection model is trained, the transformer anomaly detection model is applied to detect whether the transformer is in a normal state or an abnormal state.

[0249] Although the present invention has been described above with reference to the embodiments, various modifications may be made thereto and parts thereof may be replaced by equivalents without departing from the scope of the present invention. In particular, as long as there is no structural conflict, the various features in the embodiments disclosed in the present invention may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A transformer anomaly detection method based on multimodal deep learning, characterized in that: The specific steps are as follows: S1: Collect transformer multimodal data and perform preprocessing to obtain transformer multimodal data set D; S2: Build a transformer anomaly detection model, input transformer multimodal data into the transformer anomaly detection model, and the transformer anomaly detection model is used to detect whether the transformer is in a normal state or an abnormal state; S3: training the transformer anomaly detection model based on the constructed transformer multimodal dataset to obtain a trained transformer anomaly detection model; S4: After the transformer anomaly detection model is trained, the transformer anomaly detection model is applied to detect whether the transformer is in a normal state or an abnormal state.

2. According to claim 1, a transformer anomaly detection method based on multimodal deep learning is characterized in that: The specific process of S1 is as follows: S11: Collect multi-modal data of transformer; S12: Preprocess each of the above-collected speech data, thermal imaging images and concentration values ​​of various gases to obtain Mel spectrum x, thermal imaging image data y and gas content data z, respectively, and form a triplet<x,y,z> ; S13: Repeat S11 and S12, and get a triple after each execution.<x,y,z> , all triplets form a transformer multimodal dataset D.

3. According to claim 2, a transformer anomaly detection method based on multimodal deep learning is characterized in that: The specific process of S11 is as follows: S111: Using a microphone in the substation to record and collect voice clips generated by the vibration of the transformer during operation, the duration of each voice clip is equal, and transformer voice data is obtained; S112: During the collection of voice data from the transformer, a thermal imaging camera is used to take a picture to obtain a thermal imaging picture of the transformer; S113: Using a gas sensor, obtaining a concentration value of each gas dissolved in the transformer oil during the period of collecting voice data, wherein the gas dissolved in the transformer oil includes six gases, namely H2, CH4, C2H6, C2H4, C2H2, and CO.

4. According to claim 2, a transformer anomaly detection method based on multimodal deep learning is characterized in that: The specific process of S12 is as follows: S121: Perform a series of processing such as pre-emphasis, framing, windowing, fast Fourier transform, and Mel filter filtering on the speech data to obtain a Mel spectrum graph x; S122: performing noise reduction processing on the thermal imaging image, so that the thermal imaging image can remove salt and pepper noise and random noise while retaining edge features, thereby obtaining thermal imaging image data y; S123: First, the units of the concentration values ​​of various gases are unified, and then the total concentration O of the six gases is calculated. Among them, B i represents the concentration value of the i-th gas, that is, B1 represents the concentration value of H2, B2 represents the concentration value of CH4, B3 represents the concentration value of C2H6, B4 represents the concentration value of C2H4, B5 represents the concentration value of C2H2, and B6 represents the concentration value of CO; Then, calculate the percentage of each gas in the total concentration. The specific calculation formula is as follows: Among them, P u represents the percentage of the u-th gas in O, that is, P1 represents the percentage of H2 in O, P2 represents the percentage of CH4 in O, P3 represents the percentage of C2H6 in O, P4 represents the percentage of C2H4 in O, P5 represents the percentage of C2H2 in O, and P6 represents the percentage of CO in O; Finally, P1, P2, P3, P4, P5 and P6 constitute the gas content data z = [P1, P2, P3, P4, P5, P6]; S124: The Mel spectrum x, thermal imaging image data y and gas content data z during transformer operation are combined to form a triplet<x,y,z> .

5. The transformer anomaly detection method based on multimodal deep learning according to claim 1 is characterized in that: The specific process of S2 is as follows: S21: construct a Mel-spectrogram feature extraction module to obtain a Mel-spectrogram feature vector K1; S22: construct a thermal imaging picture data feature extraction module to obtain a thermal imaging picture data feature vector K2; S23: Use the Transformer network to fuse K1, K2 and z to obtain the feature vector R; S24: Use the status judgment module to determine whether the transformer is in an abnormal state.

6. The transformer anomaly detection method based on multimodal deep learning according to claim 5 is characterized in that: The specific process of S21 is as follows: S211: Input M1 to the first GCSM submodule to obtain the Mel frequency spectrum feature map Q 11 ; S212: Q 11 Input into the first AMA submodule to obtain the Mel spectrum feature map M 11 ; S213: M 11 After interleaving through n-1 GCSM submodules and n-1 AMA submodules, the transformer Mel spectrum feature map M is obtained. 1n ; S214: M 11 , M 12 , ..., M 1n Perform a Concat operation along the channel dimension to obtain the transformer Mel spectrum feature map M2; where the number of channels of M2 is nC, the frequency length of M2 is F, and the time length of M2 is T; S215: Input M2 into MobileNetV1 to obtain the Mel-spectrogram feature vector K1; The execution process of the GCSM submodule in S211 is as follows: S2111: For the input M1, where the number of channels of M1 is C, the frequency length of M1 is F, and the time length of M1 is T, firstly, the channels of M1 are grouped by a hyperparameter j to obtain the feature map sequence G = {G1, G2, ..., G j }, for any feature map G k ∈G,G k The number of channels is G k The frequency length is F, G k The time length is T; S2112: For any G k ∈G, first perform a global channel average pooling operation to obtain a matrix g with F rows and T columns k , then g k Input into the Sigmoid function for activation, and get the matrix σ with F rows and T columns k , where σ k is a dynamic weight parameter matrix, σ k Each element in is a weight coefficient automatically generated based on the input Mel spectrum feature map; S2113: For σ k With G k Perform Conv1D operation to obtain feature map G k1 , where G k1 The number of channels is l is the channel enhancement multiple, G k1 The frequency length is still F, G k1 The length of time T; Among them, G k1 The calculation formula is as follows: Among them, · represents the dot multiplication operation, Conv1D represents the convolution operation, the first parameter is the feature map, and the second parameter is the number of channels of the output feature map; S2114: For G k1 Perform Dwconv3D operation to extract independent spatial features of a single channel and obtain the feature map G k2 , where G k2 The number of channels is G k2 The frequency length is F, G k2 The time length is T; S2115: For G k2 Perform Conv1D operation to reduce G k2 The number of channels is obtained, and the feature map G is obtained. k3 , where G k3 The number of channels is G k3 The frequency length is F, G k3 The time length is T; Among them, G k3 The calculation formula is as follows: S2116: Finally, perform Channel Shuffle operation to obtain Mel spectrum feature map Q 11 , where Q 11 The number of channels is C, Q 11 The frequency length is F, Q 11 The time length is T; The execution process of the AMA submodule in S212 is as follows: S2121: For input Q 11 , using the first SENet network, Q 11 The channel dimension of Q is enhanced to obtain the channel attention enhanced feature map C1; the second SENet network is used to 11 The frequency dimension of Q is enhanced to obtain the frequency attention enhancement feature map F1; the third SENet network is used to 11 The time dimension of the attention is enhanced to obtain the time attention enhancement feature map T1; S2122: performing a Concat operation on C1, F1 and T1 to merge C1, F1 and T1 along the channel dimension, thereby obtaining a Mel frequency spectrum feature map A1, wherein the number of channels of A1 is 3C, the frequency length of A1 is F, and the time length of A1 is T; S2123: Perform a convolution operation on A1 through a convolution layer to integrate the information of channel, frequency and time dimensions to obtain the Mel spectrum feature map M 11 , M 11 The number of channels is C, M 11 The frequency length is F, M 11 The time length is T; The specific process of S213 is as follows: S2131: M 11 Input to the second GCSM submodule to obtain the Mel spectrum feature map Q 12 ; S2132: Q 12 Input to the second AMA submodule to obtain the Mel spectrum feature map M 12 ; S2133: M 12 Input to the third GCSM submodule to obtain the Mel spectrum feature map Q 13 ; S2134: Q 13 Input to the third AMA submodule to obtain the Mel spectrum feature map M 13 ; Until the Mel spectrum feature map Q is obtained by the nth GCSM submodule 1n ; S2135: Q 1n Input to the nth AMA submodule to obtain the transformer Mel spectrum feature map M 1n ; The specific process of S215 is as follows: S2151: In MobileNetVl, the basic spatial features are first extracted through 3x3 convolution, and the feature map size is reduced to obtain the feature map M 21 ; S2152: Through the deep convolution layer, M 21 Each channel of M is convolved independently to extract the spatial features of each channel and obtain the feature map M 22 ; S2153: M is transformed through 1x1 convolution 22 Integrate the features of different channels to capture cross-channel dependencies so as to extract key spatial and channel information from M2; S2154: After multiple layers of convolution operations, the extracted features are aggregated into a feature vector K1 through a fully connected layer. K1 contains the key information extracted from M2, where the extracted key information includes the frequency distribution information, energy distribution, time variation characteristics and spatial relationship of M2.

7. The transformer anomaly detection method based on multimodal deep learning according to claim 5 is characterized in that: The specific process of S22 is as follows: S221: Input Y1 into the OCA submodule to obtain the thermal imaging feature map Y2; S222: Input Y2 into the first GCSM submodule to obtain a thermal imaging feature map Y3, and then input Y3 into the second GCSM submodule to obtain a thermal imaging feature map Y4, until passing through the wth GCSM submodule to obtain a thermal imaging feature map Y w+2 , where Y w+2 The number of channels is U, Y w+2 The height is H, Y w+2 The width is W; S223: Y w+2 Input into MobileNetV1 to obtain the thermal imaging image data feature vector K2; The execution process of the OCA submodule in S221 is as follows: S2211: For the input thermal imaging feature map Y1, where the number of channels of Y1 is U, the height of Y1 is H, and the width of Y1 is W; for each channel of Y1, perform global maximum pooling and global average pooling respectively to extract global features in the spatial dimension, and finally obtain two vectors: maximum pooling result y max And the average pooling result y avg , where y max is a vector with length U, y avg is a vector with a length of U; S2212: For y max Normalize and get a vector max of length U; max Normalize and get a vector avg of length U; S2213: Input max and avg into the activation function Tswish to obtain an attention vector α of length U c : Where, · is the dot multiplication operation; is a hyperparameter with a value range of (0, 1]. The advantage of using the Swish activation function is that it avoids discontinuity near the zero point, so that the thermal imaging image data can better retain the characteristics. max and w avg is a learnable weight parameter that can be set by the initial weight w max >w avg , giving max a larger weight, making the weight of the global maximum pooling contribute more significantly to the OCA submodule; allowing the OCA submodule to pay attention to the most significant features in each channel, while also paying attention to the global statistical information of each channel; S2214: α c Perform element-by-element multiplication with Y1 to obtain the thermal imaging feature map Y2, where the number of channels of Y2 is U, the height of Y2 is H, and the width of Y2 is W; Y2=Y1⊙α c Among them, ⊙ represents the element-by-element multiplication operation, that is, the broadcast mechanism converts α c Expanded to the dimension of y, finally, the thermal imaging feature map Y2 enhanced by the attention mechanism is obtained.

8. The transformer anomaly detection method based on multimodal deep learning according to claim 5 is characterized in that: The specific process of S23 is as follows: S231: Input K1, K2 and z to the Transformer network. First, feature encode K1, K2 and z in the encoding layer of the Transformer network to obtain the feature vector K 11 , K 22 with z1; S232: K 11 , K 22 and z1 are input into the self-attention mechanism module, K 11 , K 22 and z1 will generate their own query, key, and value vectors through linear transformation; 11 , get the query vector q1, key vector k1 and value vector v1; for K 22 , we get the query vector q2, key vector k2 and value vector v2; for z1, we get the query vector q3, key vector k3 and value vector v3; S233: Based on K 11 , K 22 Calculate K with the query, key and value vectors of z1 11 , K 22 The correlation with z1 gives K 11 , K 22 and the weighted value vector of z1, representing v 11 , v 22 , v 33 , the calculation formula is as follows: in, is the transpose of the s-th key vector, is the dimension of the sth key vector; S234: V 11 、v 22 and v 33 Input to the decoding layer of the Transformer network for feature decoding, v 11 After decoding, we get v d11 ; for v 22 After decoding, we get v d22 , v 33 After decoding, we get v d33 ; S235: V d11 、v d22 and v d33 Perform concatenation operation to obtain a feature vector r = [v d11 , v d22 , v d33 ], r is a feature representation that integrates all modal information; S236: r is input into a feedforward neural network for processing to obtain a feature vector R, wherein the feedforward neural network is composed of several fully connected layers and activation function ReLU, which further optimizes the representation of the features; R integrates the key information of K1, K2 and z, wherein the key information includes the association information between speech and temperature, the association information between temperature and gas, and the association information between gas and speech.

9. The transformer anomaly detection method based on multimodal deep learning according to claim 5 is characterized in that: The specific process of S24 is as follows: S241: Using a state judgment module to detect the state of the transformer during operation, wherein the state judgment module is a fully connected neural network; S242: Input R to FNN, perform nonlinear mapping on the features through the hidden layer to extract features; use multiple hidden layers to extract higher-level features; the output layer receives the output of the hidden layer, performs a full connection operation and uses the Sigmoid activation function to determine whether the transformer is in an abnormal state.

10. The transformer anomaly detection method based on multimodal deep learning according to claim 1 is characterized in that: The specific process of S3 is as follows: S31: In the training process of the transformer abnormal state detection model, first initialize the parameters of all models and set the relevant hyperparameters, including n in the Mel spectrum feature extraction module, w in the thermal imaging image data feature extraction module, training rounds, batch size, optimizer selection, and learning rate; S32: Divide the transformer multimodal data set into a training set, a validation set, and a test set according to a certain ratio to ensure that the three do not overlap with each other; then, divide the training set and the validation set data into multiple batches, input a batch of training data into the model for training each time, and calculate the training loss value of the batch; S33: After all batches of the entire training set have completed a round of training, the validation set data is input into the model in batches to obtain the corresponding validation set loss value; the validation set loss value is used to monitor whether the model is overfitting and adjust the training strategy as needed; S34: Compare the training results of the transformer anomaly detection model under the same training rounds, batch size, optimizer selection, and learning rate, and finally determine the values ​​of n and w.

Images