A transformer anomaly detection method based on multi-modal deep learning

Abstract

The application discloses the technical field of transformer anomaly detection, and particularly relates to a transformer anomaly detection method based on multi-modal deep learning, which comprises the following specific steps: collecting multi-modal data of a transformer, and performing pretreatment to obtain a transformer multi-modal data set D; a transformer anomaly detection model is constructed, the multi-modal data of the transformer is input into the transformer anomaly detection model, and the transformer anomaly detection model is used to detect whether the transformer is in a normal state or an abnormal state; the transformer anomaly detection model is trained based on the constructed transformer multi-modal data set, and a trained transformer anomaly detection model is obtained; after the training of the transformer anomaly detection model is completed, the transformer anomaly detection model is applied to detect whether the transformer is in the normal state or the abnormal state. The transformer anomaly detection model is used to comprehensively perceive the running state of the transformer from multiple dimensions, reduces false positives and false negatives, and avoids the mistakes of manual inspection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of transformer anomaly detection, in particular to a transformer anomaly detection method based on multi-modal deep learning. BACKGROUND

[0002] Transformers are crucial in power systems. They can raise or lower voltage to meet different power needs, improve power transmission efficiency, and reduce transmission line losses. By stepping up, the same power can be transmitted at a lower current, reducing resistance losses. At the same time, transformers provide electrical isolation, ensuring the safety of equipment and personnel, and can stabilize voltage, reducing the impact of voltage fluctuations on power systems and electrical equipment.

[0003] As a "transit station" of the power transmission and distribution system, the stable operation of the transformer is of great significance. Therefore, early detection of potential hazards and faults in operation, and reasonable development of maintenance strategies according to the actual operation state, can effectively prevent faults, prolong the service life of equipment, and minimize economic risks caused by transformer aging or failure. However, the traditional and widely used method of manually inspecting transformers has many shortcomings. First, it is inefficient and consumes a lot of time and manpower. Second, it poses a safety risk, as inspectors work in high-voltage environments and are vulnerable to electric shock and equipment operation hazards. Third, its accuracy is limited by individual experience and skills, and may be subject to human error or misjudgment.

[0004] In addition, single-dimensional transformer anomaly detection methods have obvious shortcomings. Such methods cannot comprehensively assess the health of the transformer and are prone to misjudgment due to environmental fluctuations or temporary changes. Moreover, single-dimensional detection may lead to false positives and make it difficult to respond comprehensively and objectively to faults.

[0005] In summary, traditional anomaly detection methods have the following shortcomings:

[0006] 1. The traditional manual inspection method is costly and highly influenced by the environment;

[0007] 2. Single-modal deep learning methods cannot comprehensively assess the health of the transformer and are prone to misjudgment due to environmental fluctuations or temporary changes;

[0008] Therefore, the present application aims to develop a transformer anomaly detection method based on multi-modal deep learning, which aims to fuse multiple sensor data (such as temperature, sound, oil and gas content, etc.) and use deep learning technology to efficiently process and analyze these multi-modal data, achieving comprehensive perception and anomaly detection of transformer operating states, where the transformer operating state is divided into normal and abnormal states. SUMMARY

[0009] To solve the above technical problems, according to one aspect of the present application, the present application provides the following technical solutions:

[0010] A transformer abnormality detection method based on multi-modal deep learning, comprising the following specific steps:

[0011] S1: Collecting transformer multi-modal data and pre-processing to obtain a transformer multi-modal data set D;

[0012] S2: Constructing a transformer abnormality detection model, inputting the transformer multi-modal data into the transformer abnormality detection model, and using the transformer abnormality detection model to detect whether the transformer is in a normal state or an abnormal state;

[0013] S3: Training the transformer abnormality detection model based on the constructed transformer multi-modal data set to obtain a trained transformer abnormality detection model;

[0014] S4: After the transformer abnormality detection model is trained, applying the transformer abnormality detection model to detect whether the transformer is in a normal state or an abnormal state.

[0015] As a preferred scheme of the transformer abnormality detection method based on multi-modal deep learning, the specific process of S1 is as follows:

[0016] S11: Collecting transformer multi-modal data;

[0017] S12: Pre-processing each piece of collected voice data, thermal imaging picture and concentration value of various gases to obtain a mel spectrum x, thermal imaging picture data y and gas content data z respectively, and forming a triple <x, y, z>;

[0018] S13: Repeating S11 and S12, and obtaining a triple <x, y, z> after each execution, and forming a transformer multi-modal data set D from all triples.

[0019] As a preferred scheme of the transformer abnormality detection method based on multi-modal deep learning, the specific process of S11 is as follows:

[0020] S111: Using a microphone to record voice fragments generated by vibration of the transformer during operation in the transformer substation and collecting, the duration of each voice being equal, to obtain transformer voice data;

[0021] S112: Using a thermal imaging camera to take pictures during collection of voice data of the transformer to obtain thermal imaging pictures of the transformer;

[0022] S113: using the gas sensor, obtaining the concentration value of each dissolved gas in the transformer oil during the collection of voice data, wherein the dissolved gas in the transformer oil includes H2, CH4, C2H6, C2H4, C2H2 and CO.

[0023] As a preferred scheme of the transformer abnormality detection method based on multi-modal deep learning, the specific process of S12 is as follows:

[0024] S121: pre-emphasizing, framing, windowing, fast Fourier transform and Mel filter filtering a series of processes are performed on the voice data to obtain a Mel spectrum diagram x.

[0025] S122: the thermal imaging picture is subjected to noise reduction processing, so that the thermal imaging picture can remove salt and pepper noise and random noise while retaining edge features, to obtain thermal imaging picture data y.

[0026] S123: first, the units of various gas concentration values are unified, and then the total concentration O of the six gases is calculated,

[0027]

[0028] Wherein, B i represents the concentration value of the i-th gas, that is, B1 represents the concentration value of H2, B2 represents the concentration value of CH4, B3 represents the concentration value of C2H6, B4 represents the concentration value of C2H4, B5 represents the concentration value of C2H2, and B6 represents the concentration value of CO;

[0029] Then, the percentage of each gas in the total concentration is calculated, and the specific calculation formula is as follows:

[0030]

[0031] Wherein, P u represents the percentage of the u-th gas in O, that is, P1 represents the percentage of H2 in O, P2 represents the percentage of CH4 in O, P3 represents the percentage of C2H6 in O, P4 represents the percentage of C2H4 in O, P5 represents the percentage of C2H2 in O, and P6 represents the percentage of CO in O;

[0032] Finally, P1, P2, P3, P4, P5 and P6 constitute the gas content data z = [P1, P2, P3, P4, P5, P6];

[0033] S124: the Mel spectrum diagram x, the thermal imaging picture data y and the gas content data z during the operation of the transformer are jointly constituted into a triple <x, y, z>.

[0034] As a preferred scheme of the transformer abnormality detection method based on multi-modal deep learning, the specific process of S2 is as follows:

[0035] S21: a mel-spectrogram feature extraction module is constructed to obtain a mel-spectrogram feature vector K1;

[0036] S22: a thermal imaging picture data feature extraction module is constructed to obtain a thermal imaging picture data feature vector K2;

[0037] S23: the K1, K2 and z are fused by using a Transformer network to obtain a feature vector R;

[0038] S24: a state judgment module is used to determine whether the transformer is in an abnormal state.

[0039] As a preferred scheme of the transformer abnormality detection method based on multi-modal deep learning, the specific process of S21 is as follows:

[0040] S211: M1 is input into a first GCSM submodule to obtain a mel-spectrum feature map Q 11 ;

[0041] S212: Q 11 is input into a first AMA submodule to obtain a mel-spectrum feature map M 11 ;

[0042] S213: M 11 is alternately passed through n-1 GCSM submodules and n-1 AMA submodules to obtain a transformer mel-spectrum feature map M 1n ;

[0043] S214: M 11 , M 12 ,..., and M 1n are subjected to a Concat operation along a channel dimension to obtain a transformer mel-spectrum feature map M2; wherein the channel number of M2 is nC, the frequency length of M2 is F, and the time length of M2 is T;

[0044] S215: M2 is input into MobileNetV1 to obtain a mel-spectrum feature vector K1;

[0045] The execution process of the GCSM submodule in S211 is as follows:

[0046] S2111: for the input M1, wherein the channel number of M1 is C, the frequency length of M1 is F, and the time length of M1 is T, first, a hyperparameter j is used to group the channels of M1 to obtain a feature map sequence G={G1, G2,..., Gn}j}, for any one feature map G k ∈ G, G k has a channel number of G k has a frequency length of F, G k has a time length of T;

[0047] S2112: for any one G k ∈ G, first perform a global channel average pooling operation to obtain a matrix g k , then input g k into the Sigmoid function for activation to obtain a matrix σ k , where σ k is a dynamic weight parameter matrix, and each element in σ k is a weight coefficient automatically generated based on the input mel spectrum feature map;

[0048] S2113: perform Conv1D operation on σ k and G k to obtain a feature map G k1 , where G k1 has a channel number of l is the channel lifting multiple, the frequency length of G k1 is still F, and the time length of G k1 is T;

[0049] wherein, the calculation formula of G k1 is as follows:

[0050]

[0051] wherein, · represents dot multiplication operation, Conv1D represents performing convolution operation, the first parameter is a feature map, and the second parameter is the channel number of the output feature map;

[0052] S2114: perform Dwconv3D operation on G k1 to independently extract spatial features for a single channel to obtain a feature map G k2 , where G k2 has a channel number of G k2 has a frequency length of F, and G k2 has a time length of T;

[0053] S2115: perform Conv1D operation on G k2 to reduce the channel number of G k2 to obtain a feature map G k3 , where G k3 has a channel number of Gk3 The frequency length is F, G k3 The time length is T;

[0054] Among them, G k3 The calculation formula is as follows:

[0055]

[0056] S2116: Finally, perform the Channel Shuffle operation to obtain the Mel-spectrum feature map Q. 11 Q 11 The number of channels is C, Q 11 The frequency length is F, Q 11 The time length is T;

[0057] The execution process of the AMA submodule in S212 is as follows:

[0058] S2121: For the input Q 11 Using the first SENet network, Q 11 Attention enhancement is performed on the channel dimension to obtain the channel attention enhancement feature map C1; the second SENet network is then used to enhance the Q-axis attention. 11 Attention enhancement is applied to the frequency dimension to obtain the frequency attention-enhanced feature map F1; the third SENet network is then used to enhance the Q-axis. 11 Attention enhancement is performed on the time dimension to obtain the time attention enhancement feature map T1;

[0059] S2122: The Concat operation is used to merge C1, F1 and T1 along the channel dimension to obtain the Mel spectrum feature map A1, where the number of channels of A1 is 3C, the frequency length of A1 is F, and the time length of A1 is T.

[0060] S2123: A1 is convolved by a convolutional layer to integrate information from the channel, frequency, and time dimensions, resulting in the Mel-spectral feature map M. 11 M 11 The number of channels is C, M 11 The frequency length is F, M 11 The time length is T;

[0061] The specific process of S213 is as follows:

[0062] S2131: M 11 The input is fed into the second GCSM submodule to obtain the Mel spectrum feature map Q. 12 ;

[0063] S2132: Q 12The input is fed into the second AMA submodule to obtain the Mel spectrum feature map M. 12 ;

[0064] S2133: M 12 The input is fed into the third GCSM submodule to obtain the Mel spectrum feature map Q. 13 ;

[0065] S2134: Q 13 The input is fed into the third AMA submodule to obtain the Mel spectrum feature map M. 13 ; until the Mel spectrum feature map Q is obtained by the nth GCSM submodule. 1n ;

[0066] S2135: Q 1n The input is fed into the nth AMA submodule to obtain the transformer Mel-frequency spectrum characteristic map M. 1n ;

[0067] The specific process of S215 is as follows:

[0068] S2151: In MobileNetV1, basic spatial features are first extracted through 3x3 convolution, and the feature map size is reduced to obtain feature map M. 21 ;

[0069] S2152: Through deep convolutional layers, M 21 Each channel is convolved independently to extract the spatial features of each channel, resulting in a feature map M. 22 ;

[0070] S2153: M is convolved using a 1x1 convolution. 22 The characteristics of different channels are integrated to capture cross-channel dependencies, so as to extract key spatial and channel information from M2.

[0071] S2154: After multiple convolutional operations, the extracted features are aggregated into a feature vector K1 through a fully connected layer. K1 contains key information extracted from M2, including the frequency distribution, energy distribution, temporal variation characteristics, and spatial relationships of M2.

[0072] As a preferred embodiment of the transformer anomaly detection method based on multimodal deep learning described in this invention, the specific process of step S22 is as follows:

[0073] S221: Input Y1 into the OCA submodule to obtain the thermal imaging feature map Y2;

[0074] S222: input Y2 into the first GCSM submodule to obtain a thermal imaging feature map Y3, and then input Y3 into the second GCSM submodule to obtain a thermal imaging feature map Y4, until after the wth GCSM submodule, a thermal imaging feature map Y is obtained w+2 , wherein Y w+2 has a channel number of U, Y w+2 has a height of H, and Y w+2 has a width of W;

[0075] S223: input Y w+2 into MobileNetV1 to obtain a thermal imaging picture data feature vector K2;

[0076] The execution process of the OCA submodule in S221 is as follows:

[0077] S2211: for the input thermal imaging feature map Y1, wherein Y1 has a channel number of U, Y1 has a height of H, and Y1 has a width of W; for each channel of Y1, global maximum pooling and global average pooling are respectively performed to extract global features in the spatial dimension, and finally two vectors are obtained: a maximum pooling result y max and an average pooling result y avg , wherein y max is a vector with a length of U, and y avg is a vector with a length of U;

[0078] S2212: normalize y max to obtain a vector max with a length of U; and normalize y max to obtain a vector avg with a length of U;

[0079] S2213: input max and avg into an activation function Tswish to obtain an attention vector a c :

[0080]

[0081] wherein · is a dot product operation; is a hyperparameter, and the value range is (0, 1]; the use of the Swish activation function avoids discontinuity near zero, so that the thermal imaging picture data can better retain features, w max and w avg are learnable weight parameters, which can be set by initial weights w max > w avg, the max is given a greater weight, so that the weight of the global maximum pooling in the OCA submodule is more significant; let the OCA submodule pay attention to the most significant features in each channel, while also paying attention to the global statistical information of each channel;

[0082] S2214: alpha c is multiplied element by element with Y1 to obtain a thermal imaging feature map Y2, wherein the number of channels of Y2 is U, the height of Y2 is H, and the width of Y2 is W;

[0083] Y2=Y1⊙alpha c

[0084] wherein ⊙ represents an element-by-element multiplication operation, that is, the broadcasting mechanism expands alpha c to the dimension of y, and finally, the thermal imaging feature map Y2 enhanced by the attention mechanism is obtained.

[0085] As a preferred scheme of the transformer abnormality detection method based on multi-modal deep learning, the specific process of S23 is as follows:

[0086] S231: input K1, K2 and z into the Transformer network, first encode K1, K2 and z in the encoding layer of the Transformer network to obtain feature vectors K 11 , K 22 and z1;

[0087] S232: input K 11 , K 22 and z1 into the self-attention mechanism module, K 11 , K 22 and z1 will generate their own query, key and value vectors through linear transformation; wherein for K 11 , the query vector q1, the key vector k1 and the value vector v1 are obtained; for K 22 , the query vector q2, the key vector k2 and the value vector v2 are obtained; for z1, the query vector q3, the key vector k3 and the value vector v3 are obtained;

[0088] S233: according to the query, key and value vectors of K 11 , K 22 and z1, the relevance of K 11 , K 22 and z1 is calculated to obtain the weighted value vectors of K 11 , K 22 and z1, respectively represented as v 11 , v 22 , v 33 , and the calculation formula is as follows:

[0089]

[0090]

[0091] wherein, is the transpose of the s-th key vector, is the dimension of the s-th key vector;

[0092] S234: input v 11 , v 22 and v 33 to the decoding layer in the Transformer network for feature decoding, and obtain v 11 after decoding v d11 ; obtain v 22 after decoding v d22 , and obtain v 33 after decoding v d33 ;

[0093] S235: perform a splicing operation on v d11 , v d22 and v d33 to obtain a feature vector r = [v d11 , v d22 , v d33 ], and r is a feature representation that integrates all modal information.

[0094] S236: input r to a feedforward neural network for processing to obtain a feature vector R, wherein the feedforward neural network is composed of several fully connected layers and an activation function ReLU, which further optimizes the representation of the features; R fuses the key information of K1, K2 and z, wherein the key information includes the association information of voice and temperature, the association information of temperature and gas, and the association information of gas and voice.

[0095] As a preferred scheme of the transformer abnormality detection method based on multi-modal deep learning, the specific process of S24 is as follows:

[0096] S241: use a state judgment module to detect the state of the transformer during operation, wherein the state judgment module is a fully connected neural network.

[0097] S242: input R to FNN, perform nonlinear mapping of the features through the hidden layer, and perform feature extraction; use multiple hidden layers to extract higher-level features; the output layer receives the output of the hidden layer, performs full connection operation and uses the Sigmoid activation function, and obtains whether the transformer is in an abnormal state.

[0098] As a preferred scheme of the transformer abnormality detection method based on a multi-modal deep learning method, in the specific process of the S3, the following steps are included.

[0099] In the training process of the transformer abnormality state detection model, the parameters of all models are initialized first, and the related hyperparameters are set, including n in the mel spectrum feature extraction module, w in the thermal imaging picture data feature extraction module, training rounds, batch size, optimizer selection, and learning rate.

[0100] In the training process of the transformer abnormality state detection model, the parameters of all models are initialized first, and the related hyperparameters are set, including n in the mel spectrum feature extraction module, w in the thermal imaging picture data feature extraction module, training rounds, batch size, optimizer selection, and learning rate.

[0101] After one round of training of all batches of the entire training set is completed, the validation set data is input into the model in batches to obtain the corresponding validation set loss value. The validation set loss value is used to monitor whether the model is overfitting, and the training strategy is adjusted as needed.

[0102] Under the condition that the training rounds, batch size, optimizer selection, and learning rate are equal, the training effect of the transformer abnormality detection model is compared, and finally the values of n and w are determined.

[0103] Compared with the prior art, the following advantages are achieved:

[0104] 1. The transformer abnormality detection model is used to comprehensively perceive the running state of the transformer from multiple dimensions by fusing multiple sensor data (such as voice, temperature, and oil gas content), thereby reducing false positives and false negatives. At the same time, problems caused by insufficient experience or human errors in manual inspection are avoided.

[0105] 2. The fusion of multi-modal data enhances the adaptability of the model to environmental fluctuations and temporary changes, realizes accurate judgment of the transformer state, and thus guarantees the safe and stable operation of the power system, prolongs the service life of the equipment, and reduces economic losses caused by faults. BRIEF DESCRIPTION OF DRAWINGS

[0106] Figure 1 The flowchart of the present application is shown in the figure.

[0107] Figure 2 The schematic diagram of the transformer abnormality state detection model of the present application is shown in the figure.

[0108] Figure 3 The schematic diagram of the mel spectrum feature extraction module of the present application is shown in the figure.

[0109] Figure 4 The schematic diagram of the GCSM sub-module of the present application is shown in the figure.

[0110] Figure 5 The schematic diagram of the AMA sub-module of the present application is shown in the figure;

[0111] Figure 6 The schematic diagram of the thermal imaging picture data feature extraction module of the present application is shown in the figure;

[0112] Figure 7 The schematic diagram of the OCA sub-module of the present application is shown in the figure. DETAILED DESCRIPTION

[0113] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

[0114] The present application provides a transformer anomaly detection method based on multi-modal deep learning, please refer to Figures 1-7 , including the specific steps as follows:

[0115] S1: Collecting transformer multi-modal data and pre-processing to obtain a transformer multi-modal data set D; collecting voice data, thermal imaging pictures and concentration values of various gases dissolved in oil generated by transformer vibration in the same time period for transformers in normal state and abnormal state respectively, then pre-processing the multi-modal raw data to obtain a transformer multi-modal data set D, which is specific as follows:

[0116] The specific process of S1 is as follows:

[0117] S11: Collecting transformer multi-modal data;

[0118] The specific process of S11 is as follows:

[0119] S111: Using a microphone to record the voice fragments generated by transformer vibration during operation in the substation and collect, the duration of each voice is equal, to obtain transformer voice data;

[0120] S112: Using a thermal imaging camera to take pictures during the collection of voice data of the transformer to obtain thermal imaging pictures of the transformer;

[0121] S113: Using a gas sensor to obtain the concentration values of each transformer oil dissolved gas during the collection of voice data, wherein the transformer oil dissolved gas includes H2, CH4, C2H6, C2H4, C2H2 and CO;

[0122] including but not limited to the following embodiments:

[0123] A microphone is installed at the support structure or foundation of the transformer, and then the microphone collects voice fragments generated by vibration during operation of the transformer, each voice fragment being 20 seconds in length; during the collection of voice data, a professional thermal imaging camera is used to take thermal imaging pictures with a picture size of 512x512, the thermal imaging camera being fixed on the front of the transformer at a distance of 5 m from the transformer; finally, a gas sensor is used to view the concentration values of H2, CH4, C2H6, C2H4, C2H2 and CO in the dissolved gases in the transformer oil recorded by the gas sensor during the collection of voice data;

[0124] S12: Preprocessing each of the collected voice data, thermal imaging pictures and concentration values of various gases to obtain a mel spectrum x, thermal imaging picture data y and gas content data z respectively, and forming a triple <x, y, z>;

[0125] The specific process of S12 is as follows:

[0126] S121: Pre-emphasis, framing, windowing, fast Fourier transform, mel filter filtering and a series of processing are performed on the voice data to obtain a mel spectrum x;

[0127] Among them: converting voice data into a mel spectrum is a common operation in the field of voice processing, and is not the innovation point of the present application;

[0128] Mel spectrum: The mel spectrum is a feature representation that converts voice data into a spectrum and maps it to a mel scale, and is widely used in voice processing and voice analysis; the size of the mel spectrum is determined by the number of channels, the frequency length and the time length, wherein the number of channels is obtained by the number of sound channels of the voice or by further processing or combining the information of different frequency bands by a mel filter bank, without unit; the frequency length reflects the number of divided mel frequency bands, which is determined by the number of mel filter banks, for example, 40 or 64, without unit; the time length is the number of discrete frames in the time dimension, without unit;

[0129] S122: The thermal imaging pictures are subjected to noise reduction processing so that the thermal imaging pictures can remove salt and pepper noise and random noise while retaining edge features to obtain thermal imaging picture data y;

[0130] Including but not limited to the following embodiments:

[0131] When median filtering is used to reduce noise of the thermal imaging pictures, the median filtering covers the image by sliding pixel by pixel through a fixed size window (such as 3x3), extracts all pixel values in the window each time, sorts them by size, takes the middle value as the median, and then replaces the original value of the center pixel in the window with the median; the median filtering can effectively remove salt and pepper noise and random noise while retaining edge features;

[0132] S123: First, the units of various gas concentration values are unified, and then the total concentration O of the six gases is calculated,

[0133]

[0134] wherein B i represents the concentration value of the i-th gas, that is, B1 represents the concentration value of H2, B2 represents the concentration value of CH4, B3 represents the concentration value of C2H6, B4 represents the concentration value of C2H4, B5 represents the concentration value of C2H2, and B6 represents the concentration value of CO;

[0135] Then, the percentage of each gas in the total concentration is calculated, and the specific calculation formula is as follows:

[0136]

[0137] wherein P u represents the percentage of the u-th gas in O, that is, P1 represents the percentage of H2 in O, P2 represents the percentage of CH4 in O, P3 represents the percentage of C2H6 in O, P4 represents the percentage of C2H4 in O, P5 represents the percentage of C2H2 in O, and P6 represents the percentage of CO in O;

[0138] Finally, P1, P2, P3, P4, P5 and P6 constitute the gas content data z = [P1, P2, P3, P4, P5, P6];

[0139] including but not limited to the following embodiments:

[0140] After the units of various gas concentration values are unified, the concentration value of H2 is 120 ppm, the concentration value of CH4 is 80 ppm, the concentration value of C2H6 is 45 ppm, the concentration value of C2H4 is 30 ppm, the concentration value of C2H2 is 1 ppm, and the concentration value of CO is 300 ppm; Adding these gas concentrations, the total concentration O is 576 ppm; ppm (parts per million) is a unit for indicating the concentration of a substance, and is usually used to describe the proportion of a certain component of a gas in the total amount; it indicates the number of units of a certain substance contained in every million units;

[0141] The percentage of each gas in the total concentration is calculated, and the percentage of H2 in O is 0.83, the percentage of CH4 in O is 13.89, the percentage of C2H6 in O is 7.81, the percentage of C2H4 in O is 5.21, the percentage of C2H2 in O is 0.17, and the percentage of CO in O is 52.08;

[0142] Finally, the gas content data z = [0.83, 13.89, 7.81, 5.21, 0.17, 52.08] was obtained;

[0143] S124: The Mel-ray spectrogram x, thermal imaging data y, and gas content data z during transformer operation are combined to form a ternary set.<x,y,z> ;

[0144] S13: Repeat S11 and S12, and each execution will yield a triplet.<x,y,z> All the triples are combined to form a transformer multimodal dataset D;

[0145] S2: Construct a transformer anomaly detection model. Input the transformer's multimodal data into the transformer anomaly detection model. The transformer anomaly detection model is used to detect whether the transformer is in a normal or abnormal state. The transformer anomaly detection model is as follows: Figure 2 As shown; for any triplet<x,y,z> ∈D, the Mel spectrogram feature extraction module is used to extract features from x to obtain the Mel spectrogram feature vector K1; the thermal imaging image data feature extraction module is used to extract features from y to obtain the thermal imaging image data feature vector K2; then the Transformer network is used to fuse K1, K2 and z to obtain the feature vector R; finally, the state judgment module is used to detect whether the transformer is in a normal state or an abnormal state.

[0146] The specific process for S2 is as follows:

[0147] S21: Construct a Mel spectrogram feature extraction module to obtain the Mel spectrogram feature vector K1;

[0148] First, the triplet<x,y,z> In the graph, x is used as the Mel-spectrum feature map M1. After inputting M1 into the first GCSM submodule, the Mel-spectrum feature map Q is obtained. 11 Q 11 The input to the first AMA submodule yields the Mel spectrum feature map M. 11 M 11 After being input into the second GCSM submodule, the Mel-spectrum feature map Q is obtained. 12 Q 12 The input to the second AMA submodule yields the Mel spectrum feature map M. 12 Until M is obtained from the nth AMA submodule. 1n Then all outputs M that have passed through the AMA submodule 11 M 12 …M 1n Perform a Concat operation to obtain the Mel spectrum feature map M2, then input M2 into MobileNetV1 to finally obtain the Mel spectrum feature vector K1;

[0149] The mel-spectrogram feature extraction module is composed of n GCSM sub-modules and n AMA sub-modules in staggered series, and then Concat and MobileNetV1 are added together, wherein n is determined in the training process of the transformer abnormality detection model, and the structure is as shown in Figure 3 ;

[0150] Concat is a feature fusion operation commonly used in deep learning models, and is widely used due to its efficient structure design and low computational complexity. By concatenating multiple feature maps in the channel dimension, information from different layers or paths is integrated together, enhancing the expression ability of the model. Concat is widely used in multi-path feature fusion and cross-layer feature connection, which can enhance the expression ability of the model.

[0151] MobileNetV1 is a widely used lightweight deep learning model, known for its efficient architecture design and low computational complexity, which can greatly reduce model parameters and computational complexity. Thanks to this design, MobileNetV1 can provide strong performance with small computational overhead on resource-constrained devices such as mobile devices, becoming a model for lightweight model design.

[0152] The following embodiments are included but not limited to:

[0153] Confirmation of n: In the training process, the value of n can be selected from {4, 6, 8, 16}, and then the optimal value of n is determined according to the comparison of the model's judgment effect on the transformer state. In this embodiment, n is first set to 8.

[0154] The mel-spectrogram M1 has 64 channels, a frequency length of 40, and a time length of 2000. First, M1 is input into the first GCSM sub-module to obtain the mel-spectrogram Q 11 , which maintains the same size and channel; then Q 11 is input into the first AMA sub-module to obtain the mel-spectrogram M 11 , and continues to pass through the second GCSM sub-module and the second AMA sub-module to obtain the mel-spectrogram Q 12 and M 12 , and so on until the eighth AMA sub-module to obtain M 18 ; all mel-spectrograms processed by the AMA sub-module M 11 , M 12 ,..., M 18 ;The splicing process yields a new Mel spectrogram M2, which has 192 channels, a frequency length of 40, and a time length of 2000. Finally, M2 is input into MobileNetV1 to obtain the Mel spectrogram feature vector K1, which has a vector length of 1000.

[0155] The specific procedures for S21 are as follows:

[0156] S211: Input M1 into the first GCSM submodule to obtain the Mel-spectrum feature map Q. 11 This invention designs a GCSM (Grouped Convolution and Shuffle Module) submodule; the GCSM submodule is used to extract features from the Mel spectrogram to obtain a feature-enhanced Mel spectrogram feature map; the structure of the GCSM submodule is as follows. Figure 4 As shown;

[0157] The execution flow of the GCSM submodule in S211 is as follows:

[0158] S2111: For the input M1, where the number of channels of M1 is C, the frequency length of M1 is F, and the time length of M1 is T, the channels of M1 are first grouped using a hyperparameter j to obtain the feature map sequence G = {G1, G2, ..., G...} j For any feature map G k ∈G, G k The number of channels is G k The frequency length is F, G k The time length is T;

[0159] S2112: For any G k For each element ∈G, a global channel average pooling operation is first performed to obtain an F-row, T-column matrix g. k Then g k The input is fed into the Sigmoid function for activation, resulting in an F-row, T-column matrix σ. k , where σ k It is a dynamic weight parameter matrix, σ k Each element in the graph is a weight coefficient automatically generated based on the input Mel-spectrum feature map;

[0160] The beneficial effects of S2112: By performing global channel average pooling, it can enhance the expressive power of channel features and achieve the effect of σ... k The real-time adjustment of the convolutional kernel weights enables the GCSM module to enhance or weaken features based on the characteristics of the input feature map.

[0161] Wherein, the global channel average pooling (GCAP) is a channel dimension pooling operation for compressing information in a multi-channel feature map into a two-dimensional space, which can retain the global features of each spatial position; it generates a two-dimensional space by taking the average of the feature values on all channels of each spatial position;

[0162] S2113: σ k is performed, and a feature map G k is obtained, wherein the channel number of G k1 is k1 l is the lifting multiple of the channel (which can be 3), the frequency length of G k1 is still F, and the time length T of G k1 ;

[0163] Wherein, the calculation formula of G k1 is as follows:

[0164]

[0165] Wherein, · represents dot multiplication, Conv1D represents performing convolution operation, and the first parameter is a feature map and the second parameter is the channel number of the output feature map;

[0166] The beneficial effect of S2113 is that the feature expression capability of G k is improved by lifting the channel number;

[0167] S2114: Dwconv3D operation is performed on G k1 to independently extract spatial features of a single channel, and a feature map G k2 is obtained, wherein the channel number of G k2 is The frequency length of G k2 is F, and the time length T of G k2 ;

[0168] The beneficial effect of S2114 is that by independently extracting spatial features of a single channel of G k1 , the unique information of each channel is retained, while the calculation amount and parameter amount are significantly reduced, higher calculation efficiency and flexibility are provided, which helps to improve the feature extraction effect and further enhance the performance of the model;

[0169] S2115: Conv1D operation is performed on G k2 to reduce the channel number of G k2 , and a feature map G k3 is obtained, wherein the channel number of G k3 is​ G k3 The frequency length of G k3 is F, and the time length of G

[0170] The beneficial effect of S2115 is that the key information is screened out by reducing the number of channels, and the redundant features are discarded, so as to improve the representation ability of the features, reduce the risk of overfitting, and enhance the stability of gradient propagation.

[0171] The calculation formula of G k3 is as follows:

[0172]

[0173] S2116: Finally, a Channel Shuffle operation is performed to obtain a mel-spectrum feature map Q 11 , wherein the number of channels of Q 11 is C, the frequency length of Q 11 is F, and the time length of Q 11 is T.

[0174] Conv1D: a one-dimensional convolution operation, whose convolution kernel slides along the time axis or sequence axis of the feature map, used to extract local features in sequence data; the convolution operation acts on all input channels and realizes cross-channel information integration through shared weights; it can efficiently capture local patterns in sequence data while preserving global context information.

[0175] Dwconv3D: a three-dimensional depthwise separable convolution operation, whose convolution kernel independently acts on each input channel to extract spatial features of a single channel of the input feature map without involving cross-channel weight sharing; it preserves the unique information of each channel of the input feature map while significantly reducing the amount of calculation and parameters; it can efficiently capture local patterns in three-dimensional data while reducing computational complexity.

[0176] Channel Shuffle: a simple and efficient channel shuffling operation used for group convolution channel interaction enhancement; by grouping, rearranging and restoring channels, Channel Shuffle can significantly alleviate the problem of information isolation in group convolution while maintaining computational efficiency.

[0177] S212: input Q 11 into a first AMA sub-module to obtain a mel-spectrum feature map M 11 ; an AMA (All Dimensions Attention) sub-module is designed; the AMA sub-module is used to enhance the features of each dimension of the mel-spectrum map to obtain a feature-enhanced mel-spectrum feature map; the structure of the AMA sub-module is as follows:Figure 5 as shown;

[0178] The execution process of the AMA submodule in S212 is as follows:

[0179] S2121: For the input Q 11 , a first SENet network is used to perform attention enhancement on the channel dimension of Q 11 to obtain a channel attention enhanced feature map C1; a second SENet network is used to perform attention enhancement on the frequency dimension of Q 11 to obtain a frequency attention enhanced feature map F1; and a third SENet network is used to perform attention enhancement on the time dimension of Q 11 to obtain a time attention enhanced feature map T1.

[0180] Beneficial effects of S2121: By using the SENet network, adaptive weight coefficients can be generated for the channel dimension, the frequency dimension and the time dimension to automatically adjust the weight of each channel, time step and frequency component; these weights reflect the importance of each dimension, thereby enhancing the attention of the mel-spectrum feature extraction module to key features.

[0181] S2122: C1, F1 and T1 are combined through Concat operation to combine C1, F1 and T1 along the channel dimension, thereby obtaining a mel-spectrum feature map A1, wherein the number of channels of A1 is 3C, the frequency length of A1 is F, and the time length of A1 is T.

[0182] Beneficial effects of S2122: Through the Concat operation, the feature information of each dimension after the SENet can be fully retained.

[0183] S2123: A convolution layer is used to perform convolution operation on A1 to integrate the information of the channel, frequency and time dimensions, thereby obtaining a mel-spectrum feature map M 11 , wherein the number of channels of M 11 is C, the frequency length of M 11 is F, and the time length of M 11 is T.

[0184] Beneficial effects of S2123: Through the convolution layer, the features of each dimension are fused, thereby optimizing the understanding and representation ability of the mel-spectrum feature extraction module for the mel-spectrum graph.

[0185] The three SENet networks in S2123 are completely identical. Figure 5

[0186] ​SENet (Squeeze-and-Excitation Network) is a network that improves the information interaction and feature expression ability between the same dimensions. By introducing the Squeeze and Excitation steps, the selected dimensions are adaptively weighted, which significantly improves the feature representation ability. Through these two steps, the network can learn the importance of each dimension, thereby enhancing the expression of key features and suppressing irrelevant features, thus improving the performance and generalization ability of the network.

[0187] S213: input M 11 to the first GCSM submodule, and obtain the mel spectrum feature map Q 1n ;

[0188] The specific process of S213 is as follows:

[0189] S2131: input M 11 to the second GCSM submodule, and obtain the mel spectrum feature map Q 12 ;

[0190] S2132: input Q 12 to the second AMA submodule, and obtain the mel spectrum feature map M 12 ;

[0191] S2133: input M 12 to the third GCSM submodule, and obtain the mel spectrum feature map Q 13 ;

[0192] S2134: input Q 13 to the third AMA submodule, and obtain the mel spectrum feature map M 13 ; until the mel spectrum feature map Q 1n is obtained by the nth GCSM submodule;

[0193] S2135: input Q 1n to the nth AMA submodule, and obtain the transformer mel spectrum feature map M 1n ;

[0194] S214: perform Concat operation on M 11 , M 12 , …, M 1n along the channel dimension, and obtain the transformer mel spectrum feature map M2; wherein the channel number of M2 is nC, the frequency length of M2 is F, and the time length of M2 is T;

[0195] S215: input M2 into MobileNetV1, and obtain the mel spectrum feature vector K1;

[0196] The specific process of S215 is as follows:

[0197] S2151: In MobileNetV1, first, basic spatial features are extracted through 3x3 convolution, and the feature map size is reduced to obtain a feature map M 21 ;

[0198] S2152: Through a deep convolutional layer, a convolution operation is independently performed on each channel of M 21 , spatial features of each channel are extracted, and a feature map M 22 is obtained.

[0199] S2153: Through 1x1 convolution, the features of different channels of M 22 are integrated to capture the cross-channel dependency, so as to extract key spatial and channel information from M2;

[0200] S2154: After a plurality of convolutional operations, the extracted features are aggregated into a feature vector K1 through a fully connected layer, K1 contains key information extracted from M2, wherein the extracted key information includes frequency distribution information, energy distribution, time variation characteristics and spatial relationship of M2;

[0201] S22: Construct a thermal imaging picture data feature extraction module to obtain a thermal imaging picture data feature vector K2;

[0202] First, y of the triple <x, y, z> is taken as a thermal imaging feature map Y1, Y1 is input into an OCA submodule to obtain a thermal imaging feature map Y2, Y2 is input into a first GCSM submodule to obtain a thermal imaging feature map Y3, Y3 is input into a second GCSM submodule to obtain a thermal imaging feature map Y4, and so on, until a thermal imaging feature map Y w+2 is obtained after the wth GCSM submodule, and finally Y w+2 is input into MobileNetV1 to obtain a thermal imaging picture data feature vector K2;

[0203] The thermal imaging picture data feature extraction module is composed of an OCA submodule, w GCSM submodules and MobileNetV1 in series, wherein w is determined in the training process of the transformer abnormality detection model, and the structure is as shown in Figure 6 ;

[0204] The specific process of S22 is as follows:

[0205] S221: Y1 is input into the OCA submodule to obtain a thermal imaging feature map Y2; the present application designs an OCA (Only Channel Attention) submodule; the channel features of Y1 are strengthened by using the OCA submodule to obtain a thermal imaging feature map Y2, wherein the OCA submodule is as shown inFigure 7 as shown;

[0206] The execution flow of the OCA submodule in S221 is as follows:

[0207] S2211: For the input thermal imaging feature map Y1, where the number of channels of Y1 is U, the height of Y1 is H, and the width of Y1 is W; for each channel of Y1, respectively perform global maximum pooling and global average pooling to extract global features in the spatial dimension, and finally obtain two vectors: the maximum pooling result y max and the average pooling result y avg , where y max is a vector with a length of U, and y avg is a vector with a length of U.

[0208] Beneficial effects of S2211: the global maximum pooling can extract the most significant features in each channel, and the global average pooling can extract the global statistical information of each channel.

[0209] where, global maximum pooling: a pooling operation that takes the maximum value of all values in the spatial dimension (i.e., height H and width W) of the input feature map, thereby compressing the two-dimensional feature map into a one-dimensional channel vector; this operation can extract the most significant features in each channel;

[0210] Global average pooling: a pooling operation that calculates the average value of all values in the spatial dimension (i.e., height H and width W) of the input feature map, thereby compressing the two-dimensional feature map into a one-dimensional channel vector; this operation can extract the global statistical information of each channel.

[0211] S2212: normalize yma x to obtain a vector max with a length of U; normalize yma x to obtain a vector avg with a length of U.

[0212] where, normalization: a data preprocessing operation that adjusts the values of the input data to the range [0, 1] to eliminate the influence of the numerical scale between different features or samples;

[0213] S2213: input max and avg into the activation function Tswish to obtain an attention vector a c with a length of U.

[0214]

[0215] where, · is the dot product operation. is a hyper-parameter, and its value interval is (0, 1]; the advantage of using the Swish activation function is that the non-continuity near zero is avoided, so that the thermal imaging picture data can better retain the features, w max and w avg are learnable weight parameters, which can be set by initial weights w max > w avg , giving max a larger weight, so that the weight of global max pooling in the OCA submodule is more significant; let the OCA submodule pay attention to the most significant features in each channel, while also paying attention to the global statistical information of each channel;

[0216] S2214: element-wise multiply a c and Y1 to obtain a thermal imaging feature map Y2, wherein the number of channels of Y2 is U, the height of Y2 is H, and the width of Y2 is W;

[0217] Y2 = Y1 O a c

[0218] wherein O represents an element-wise multiplication operation, that is, the broadcast mechanism expands a c to the dimension of y, and finally, a thermal imaging feature map Y2 enhanced by the attention mechanism is obtained;

[0219] Beneficial effects of S2214: through the OCA submodule, the thermal imaging picture data feature extraction module can pay more attention to important information between channels, improving the quality of feature expression;

[0220] wherein the broadcast mechanism: refers to a mechanism for expanding a smaller vector to match the shape of a larger tensor through certain rules when vectors of different dimensions or sizes are operated, so as to realize element-wise operation;

[0221] S222: input Y2 into the first GCSM submodule to obtain a thermal imaging feature map Y3, then input Y3 into the second GCSM submodule to obtain a thermal imaging feature map Y4, and so on until the thermal imaging feature map Y w+2 is obtained after the wth GCSM submodule, wherein the number of channels of Y w+2 is U, the height of Y w+2 is H, and the width of Y w+2 is W;

[0222] S223: input Y w+2 into MobileNetVl to obtain a thermal imaging picture data feature vector K2;

[0223] S23: use the Transformer network to fuse K1, K2 and z to obtain a feature vector R;

[0224] The specific flow of S23 is as follows:

[0225] S231: input K1, K2 and z to the Transformer network, first encode K1, K2 and z in the encoding layer of the Transformer network to obtain feature vectors K 11 , K 22 and z1;

[0226] S232: input K 11 , K 22 and z1 to the self-attention mechanism module, K 11 , K 22 and z1 will generate respective query (Query), key (Key) and value (Value) vectors through linear transformation; wherein for K 11 , the query vector q1, the key vector k1 and the value vector v1 are obtained; for K 22 , the query vector q2, the key vector k2 and the value vector v2 are obtained; for z1, the query vector q3, the key vector k3 and the value vector v3 are obtained;

[0227] S233: according to the query (Query), key (Key) and value (Value) vectors of K 11 , K 22 and z1, the relevance of K 11 , K 22 and z1 is calculated, and the weighted value vectors of K 11 , K 22 and z1 are obtained, respectively represented as v 11 , v 22 and v 33 , and the calculation formula is as follows:

[0228]

[0229]

[0230] wherein, is the transpose of the s-th key (Key) vector, is the dimension of the s-th key (Key) vector;

[0231] S234: input v 11 , v 22 and v 33 to the decoding layer in the Transformer network for feature decoding, and after decoding v 11 , v d11 is obtained; after decoding v 22 , v d22 is obtained, and after decoding v 33 , v d33 is obtained;

[0232] S235: Concatenate v d11 , v d22 and v d33 to obtain a feature vector r = [v d11 , v d22 , v d33 ], r is a feature representation that integrates all modal information;

[0233] S236: Input r to a feedforward neural network for processing to obtain a feature vector R, wherein the feedforward neural network consists of several fully connected layers and an activation function ReLU, further optimizing the representation of the features; R fuses the key information of K1, K2 and z, wherein the key information includes the association information of voice and temperature, the association information of temperature and gas, and the association information of gas and voice;

[0234] Wherein, the Transformer network is a deep learning architecture widely used in natural language processing (NLP) and other tasks, famous for its efficient self-attention mechanism and strong modeling ability; It completely abandons the traditional recurrent neural network (RNN) and convolutional neural network (CNN) structure, and instead uses the self-attention (Self-Attention) mechanism to model the long-term dependencies in sequence data; The advantage of the Transformer network in multi-modal data fusion lies in its strong self-attention mechanism, efficient cross-modal modeling ability, long-distance dependency capture, parallel computing ability, and good scalability and adaptability;

[0235] S24: Use the state judgment module to determine whether the transformer is in an abnormal state;

[0236] The specific process of S24 is as follows:

[0237] S241: Use the state judgment module to detect the state of the transformer during operation, wherein the state judgment module is a fully connected neural network (Fully Connected Neural Network, FNN);

[0238] S242: Input R to FNN, perform nonlinear mapping of features through hidden layers for feature extraction; Use multiple hidden layers to extract higher-level features; The output layer receives the output of the hidden layer, performs full connection operation and uses the Sigmoid activation function to determine whether the transformer is in an abnormal state;

[0239] Among them, the fully connected neural network (Fully Connected Neural Network, FNN) is a basic model commonly used in deep learning tasks. Each layer of neurons is connected to every neuron in the previous layer, and the weighted sum and activation function are used to realize nonlinear transformation. It can automatically learn complex input-output relationships and effectively integrate features of different modalities to form a unified representation. The fully connected network is highly adaptable and can handle multi-modal data. It optimizes parameters through backpropagation and is widely used in feature fusion and classification tasks. In binary classification tasks, the fully connected network can effectively perform binary classification through the Sigmoid activation function in the output layer, and is suitable for handling binary classification problems with clear labels.

[0240] S3: training the transformer anomaly detection model based on the constructed transformer multi-modal data set to obtain the trained transformer anomaly detection model;

[0241] The specific process of S3 is as follows:

[0242] S31: In the training process of the transformer anomaly state detection model, first, initialize all model parameters and set related hyperparameters, including n in the mel spectrum feature extraction module, w in the thermal imaging picture data feature extraction module, training epochs, batch size, optimizer selection, learning rate, etc.

[0243] S32: Divide the transformer multi-modal data set into training set, validation set and test set according to a certain proportion to ensure that the three are not overlapped. Then, divide the training set and validation set data into multiple batches, and input each batch of training data into the model for training each time to calculate the batch training loss value (loss);

[0244] S33: After completing a round of training on all batches of the entire training set, input the validation set data in batches into the model to obtain the corresponding validation set loss value (batch_loss). The validation set loss value is used to monitor whether the model has overfitting, and the training strategy is adjusted as needed, such as early termination of training or adjustment of learning rate;

[0245] S34: Under the condition that the training epochs, batch size, optimizer selection, and learning rate are equal, compare the training effect of the transformer anomaly detection model, and finally determine the values of n and w;

[0246] Including but not limited to the following embodiments:

[0247] The transformer multi-modal data set is divided into a training set, a validation set and a test set according to a certain proportion, to ensure that the three are not overlapped, wherein the training set accounts for 70% of the total number of the data set, the validation set and the test set each accounts for 15%; the training round (epoch) is set to 150, and the batch size is 8; the optimizer is selected as a stochastic gradient descent (SGD), the initial learning rate is set to 0.01, and the weight decay is set to 0.0005; if the validation set loss is not improved for 5 consecutive rounds, the learning rate is halved; when the validation set loss no longer decreases, it is considered that the model has basically converged, and the training is ended; the training effect of the transformer anomaly detection model is compared to obtain the value of n as 8 and the value of w as 4;

[0248] S4: After the transformer anomaly detection model is trained, the transformer anomaly detection model is applied to detect whether the transformer is in a normal state or an abnormal state.

[0249] Although the present application has been described with reference to the embodiments above, various improvements can be made thereto and components thereof can be substituted with equivalents without departing from the scope of the present application. In particular, features in the embodiments disclosed herein can be combined with each other in any manner as long as there is no structural conflict, and the combinations are not exhaustively described herein only for the purpose of saving space and resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A transformer anomaly detection method based on multimodal deep learning, characterized in that, The specific steps are as follows: S1: Collect transformer multimodal data and preprocess it to obtain transformer multimodal dataset D; S2: Construct a transformer anomaly detection model. Input the transformer multimodal data into the transformer anomaly detection model. The transformer anomaly detection model is used to detect whether the transformer is in a normal state or an abnormal state. S3: The transformer anomaly detection model is trained based on the constructed transformer multimodal dataset to obtain the trained transformer anomaly detection model; S4: After the transformer anomaly detection model is trained, apply the transformer anomaly detection model to detect whether the transformer is in a normal or abnormal state. The specific process of S1 is as follows: S11: Acquire multimodal data of the transformer; S12: Preprocess each segment of voice data, thermal imaging image, and concentration values ​​of various gases collected above to obtain Mel-frequency spectrogram x, thermal imaging image data y, and gas content data z, and form a triplet.<x,y,z> ; S13: Repeat S11 and S12, and each execution will yield a triplet.<x,y,z> All the triples are combined to form a transformer multimodal dataset D; The specific process of S2 is as follows: S21: Construct a Mel spectrogram feature extraction module to obtain the Mel spectrogram feature vector K1; S22: Construct a feature extraction module for thermal imaging image data to obtain the feature vector K2 of thermal imaging image data; S23: Using the Transformer network, K1, K2, and z are fused to obtain the feature vector R; S24: Use the status judgment module to determine whether the transformer is in an abnormal state; The specific process of S21 is as follows: S211: First, combine the triplet<x,y,z> x is used as the Mel-spectral feature map M1. Then, M1 is input into the first GCSM submodule to obtain the Mel-spectral feature map Q. 11 ; S212: Q 11 The input is fed into the first AMA submodule to obtain the Mel spectrum feature map M. 11 ; S213: M 11 The transformer Mel-frequency spectrum characteristic map M is obtained by interleaving through n-1 GCSM submodules and n-1 AMA submodules. 1n ; S214: M 11 M 12 ,…,M 1n Perform a Concat operation along the channel dimension to obtain the transformer Mel-frequency spectral feature map M2; where the number of channels in M2 is nC, the frequency length of M2 is F, and the time length of M2 is T. S215: Input M2 into MobileNetV1 to obtain the Mel spectrogram feature vector K1; The execution flow of the GCSM submodule in S211 is as follows: S2111: For the input M1, where the number of channels of M1 is C, the frequency length of M1 is F, and the time length of M1 is T, the channels of M1 are first grouped using a hyperparameter j to obtain the feature map sequence G = {G1, G2, ..., G...} j For any feature map G k ∈G, G k The number of channels is G k The frequency length is F, G k The time length is T; S2112: For any G k For each element ∈G, a global channel average pooling operation is first performed to obtain an F-row, T-column matrix g. k Then g k The input is fed into the Sigmoid function for activation, resulting in an F-row, T-column matrix σ. k , where σ k It is a dynamic weight parameter matrix, σ k Each element in the graph is a weight coefficient automatically generated based on the input Mel-spectrum feature map; S2113: For σ k With G k Perform Conv1D operation to obtain feature map G. k1 G k1 The number of channels is l is the boost factor for the channel, G k1 The frequency length remains F, G k1 The time length is T; Among them, G k1 The calculation formula is as follows: Where · represents dot product operation, Conv1D represents convolution operation, the first parameter is the feature map, and the second parameter is the number of channels of the output feature map; S2114: Regarding G k1 Perform Dwconv3D operations to extract spatial features independently from each channel, resulting in a feature map G. k2 G k2 The number of channels is G k2 The frequency length is F, G k2 The time length is T; S2115: Regarding G k2 Perform Conv1D operations to reduce G k2 The number of channels is used to obtain the feature map G. k3 G k3 The number of channels is G k3 The frequency length is F, G k3 The time length is T; Among them, G k3 The calculation formula is as follows: S2116: Finally, perform the Channel Shuffle operation to obtain the Mel-spectrum feature map Q. 11 Q 11 The number of channels is C, Q 11 The frequency length is F, Q 11 The time length is T; The execution process of the AMA submodule in S212 is as follows: S2121: For the input Q 11 Using the first SENet network, Q 11 Attention enhancement is performed on the channel dimension to obtain the channel attention enhancement feature map C1; the second SENet network is then used to enhance the Q-axis attention. 11 Attention enhancement is applied to the frequency dimension to obtain the frequency attention-enhanced feature map F1; the third SENet network is then used to enhance the Q-axis. 11 Attention enhancement is performed on the time dimension to obtain the time attention enhancement feature map T1; S2122: The Concat operation is used to merge C1, F1 and T1 along the channel dimension to obtain the Mel spectrum feature map A1, where the number of channels of A1 is 3C, the frequency length of A1 is F, and the time length of A1 is T. S2123: A1 is convolved by a convolutional layer to integrate information from the channel, frequency, and time dimensions, resulting in the Mel-spectral feature map M. 11 M 11 The number of channels is C, M 11 The frequency length is F, M 11 The time length is T; The specific process of S213 is as follows: S2131: M 11 The input is fed into the second GCSM submodule to obtain the Mel spectrum feature map Q. 12 ; S2132: Q 12 The input is fed into the second AMA submodule to obtain the Mel spectrum feature map M. 12 ; S2133: M 12 The input is fed into the third GCSM submodule to obtain the Mel spectrum feature map Q. 13 ; S2134: Q 13 The input is fed into the third AMA submodule to obtain the Mel spectrum feature map M. 13 ; until the Mel spectrum feature map Q is obtained by the nth GCSM submodule. 1n ; S2135: Q 1n The input is fed into the nth AMA submodule to obtain the transformer Mel-frequency spectrum characteristic map M. 1n ; The specific process of S215 is as follows: S2151: In MobileNetV1, basic spatial features are first extracted through 3x3 convolution, and the feature map size is reduced to obtain feature map M. 21 ; S2152: Through deep convolutional layers, M 21 Each channel is convolved independently to extract the spatial features of each channel, resulting in a feature map M. 22 ; S2153: M is convolved using a 1x1 convolution. 22 The characteristics of different channels are integrated to capture cross-channel dependencies, so as to extract key spatial and channel information from M2. S2154: After multiple convolutional operations, the extracted features are aggregated into a feature vector K1 through a fully connected layer. K1 contains key information extracted from M2, including the frequency distribution, energy distribution, temporal variation characteristics and spatial relationships of M2. The specific process of S22 is as follows: S221: First, combine the triplet<x,y,z> The y is used as the thermal imaging feature map Y1. Then Y1 is input into the OCA submodule to obtain the thermal imaging feature map Y2. S222: Input Y2 into the first GCSM submodule to obtain thermal imaging feature map Y3, then input Y3 into the second GCSM submodule to obtain thermal imaging feature map Y4, and so on until the w-th GCSM submodule is passed to obtain thermal imaging feature map Y. w+2 , where Y w+2 The number of channels is U, Y w+2 The height is H, Y w+2 The width is W; S223: Y w+2 Inputting it into MobileNetV1 yields the feature vector K2 of the thermal imaging image data; The execution flow of the OCA submodule in S221 is as follows: S2211: For the input thermal imaging feature map Y1, where the number of channels in Y1 is U, the height of Y1 is H, and the width of Y1 is W; for each channel of Y1, perform global max pooling and global average pooling respectively to extract global features in the spatial dimension, finally obtaining two vectors: the max pooling result y max and average pooling result y avg , where y max It is a vector with length U and y avg It is a vector with length U; S2212: For y max Normalization yields a vector max of length U; for y max Normalization yields a vector avg of length U; S2213: Input max and avg into the activation function Tswish to obtain an attention vector α of length U. c : Where · represents the dot product operation; The hyperparameter is defined as w, with values ​​ranging from (0,1). The advantage of using the Swish activation function is that it avoids discontinuities near zero, allowing thermal imaging data to better preserve features. max and w avg These are learnable weight parameters, which can be set by initial weights w. max >w avg This assigns a larger weight to max, making the contribution of the global max pooling weights more significant in the OCA submodule; it allows the OCA submodule to notice the most salient features in each channel, while also paying attention to the global statistics of each channel. S2214: α c Element-wise multiplication with Y1 yields thermal imaging feature map Y2, where the number of channels in Y2 is U, the height of Y2 is H, and the width of Y2 is W. Y2=Y1⊙α c Here, ⊙ represents element-wise multiplication, i.e., the broadcast mechanism multiplies α. c Extending to the dimension of y, we finally obtain the thermal imaging feature map Y2 enhanced by the attention mechanism.

2. The transformer anomaly detection method based on multimodal deep learning according to claim 1, characterized in that, The specific process of S11 is as follows: S111: Use a microphone inside the substation to record and collect voice segments generated by the vibration of the transformer during operation. Each voice segment has an equal duration to obtain transformer voice data. S112: During the acquisition of voice data from the transformer, a thermal imaging camera is used to take pictures to obtain thermal images of the transformer; S113: Using a gas sensor, obtain the concentration value of each dissolved gas in the transformer oil during the acquisition of voice data, wherein the dissolved gases in the transformer oil include six gases: H2, CH4, C2H6, C2H4, C2H2, and CO.

3. The transformer anomaly detection method based on multimodal deep learning according to claim 1, characterized in that, The specific process of S12 is as follows: S121: Perform a series of processing steps on the speech data, including pre-emphasis, framing, windowing, fast Fourier transform, and Mel filter filtering, to obtain the Mel spectrogram x; S122: Denoise the thermal imaging image so that salt-and-pepper noise and random noise are removed while edge features are preserved, to obtain thermal imaging image data y. S123: First, standardize the units for the concentration values ​​of various gases, then calculate the total concentration O of the six gases. Among them, B i B1 represents the concentration of the i-th gas, B2 represents the concentration of H2, B3 represents the concentration of CH4, B4 represents the concentration of C2H6, B5 represents the concentration of C2H2, and B6 represents the concentration of CO. Then, calculate the percentage of each gas in the total concentration, using the following formula: Among them, P u P1 represents the percentage of gas u in O, P2 represents the percentage of H2 in O, P3 represents the percentage of CH4 in O, P4 represents the percentage of C2H6 in O, P5 represents the percentage of C2H2 in O, and P6 represents the percentage of CO in O. Finally, P1, P2, P3, P4, P5, and P6 are combined to form the gas content data z = [P1, P2, P3, P4, P5, P6]. S124: The Mel-ray spectrogram x, thermal imaging data y, and gas content data z during transformer operation are combined to form a ternary set.<x,y,z> .

4. The transformer anomaly detection method based on multimodal deep learning according to claim 1, characterized in that, The specific process of S23 is as follows: S231: Input K1, K2, and z into the Transformer network. First, perform feature encoding on K1, K2, and z in the encoding layer of the Transformer network to obtain the feature vector K. 11 K 22 With z1; S232: K 11 K 22 The z1 input is fed into the self-attention mechanism module, K 11 K 22 z1 and z2 will generate their respective query, key, and value vectors through a linear transformation; where for K 11 This yields the query vector q1, the key vector k1, and the value vector v1; for K 22 For z1, we obtain query vector q2, key vector k2, and value vector v2; for z1, we obtain query vector q3, key vector k3, and value vector v3. S233: According to K 11 K 22 Given the query, key, and value vectors of z1, calculate K. 11 K 22 The correlation with z1 yields K. 11 K 22 The weighted vectors of z1 and z2 respectively represent v 11 v 22 v 33 The calculation formula is as follows: in, Let be the transpose of the s-th key vector. It is the dimension of the s-th key vector; S234: v 11 v 22 and v 33 The input is fed into the decoding layer of the Transformer network for feature decoding, and v 11 After decoding, we get v d11 ; for v 22 After decoding, we get v d22 v 33 After decoding, we get v d33 ; S235: v d11 v d22 and v d33 Perform the concatenation operation to obtain a feature vector r = [v d11 ,v d22 ,v d33 ], r is a feature representation that integrates information from all modalities; S236: Input r into the feedforward neural network for processing to obtain a feature vector R, where the feedforward neural network consists of several fully connected layers and the ReLU activation function, which further optimizes the representation of features; R integrates the key information of K1, K2 and z, where the key information includes the correlation information between speech and temperature, the correlation information between temperature and gas, and the correlation information between gas and speech.

5. The transformer anomaly detection method based on multimodal deep learning according to claim 1, characterized in that, The specific process of S24 is as follows: S241: The state judgment module is used to detect the state of the transformer during operation. The state judgment module is a fully connected neural network. S242: Input R to FNN, perform non-linear mapping on features through hidden layers to extract features; use multiple hidden layers to extract higher-level features; the output layer receives the output of the hidden layers, performs a fully connected operation and uses the Sigmoid activation function to determine whether the transformer is in an abnormal state.

6. The transformer anomaly detection method based on multimodal deep learning according to claim 1, characterized in that, The specific process of S3 is as follows: S31: During the training process of the transformer abnormal state detection model, first initialize all model parameters and set relevant hyperparameters, including n in the Mel spectrogram feature extraction module, w in the thermal imaging image data feature extraction module, training rounds, batch size, optimizer selection, and learning rate. S32: Divide the transformer multimodal dataset into training set, validation set and test set according to a certain ratio to ensure that the three do not overlap; then, divide the training set and validation set data into multiple batches, and input one batch of training data into the model for training each time, and calculate the training loss value of the batch. S33: After completing one round of training across all batches of the entire training set, the validation set data is input into the model in batches to obtain the corresponding validation set loss value; the validation set loss value is used to monitor whether the model is overfitting and to adjust the training strategy as needed. S34: Under the condition that the number of training rounds, batch size, optimizer selection, and learning rate are equal, compare the training effect of the transformer anomaly detection model, and finally determine the values ​​of n and w.

Images