Transformer state parameter prediction method based on depth separable convolution and double prediction heads

Abstract

The invention discloses a transformer state parameter prediction method based on depth separable convolution and double prediction heads. The method comprises the following steps: firstly, constructing a transformer state parameter time sequence as input, directly inputting original data to a linear layer after normalization processing and data segmentation, and outputting a preliminary prediction result to capture an overall trend; the method comprises the following steps: calculating an original sequence frequency spectrum through FFT (Fast Fourier Transform) to obtain a period of the original sequence, filling and fragmenting the original sequence, transmitting converted data into a module constructed by a plurality of depth separable convolutional layers to extract features, restoring a returned value into a one-dimensional time sequence, transmitting the one-dimensional time sequence into a linear layer interspersed with a nonlinear activation function, and generating a nonlinear prediction result. And finally, adding the prediction result with the preliminary prediction result to obtain a final prediction result. Through the design, the model effectively combines linear and nonlinear information within the cycle and during the cycle, and the feature extraction efficiency and the prediction precision are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of time series prediction, and in particular to a transformer state parameter prediction method based on deep separable convolution and dual prediction heads, which realizes transformer state parameter data modeling and prediction with a long time span through fast Fourier transform, deep separable convolution, Inception_Block_v1 and dual-head prediction mechanism. Background Art

[0002] The temperature of power transformers is usually affected by factors such as transformer load. Accurately predicting the temperature change of power transformers is of great significance for the safe operation of power systems, equipment maintenance, and fault warning. Due to the locality of the one-dimensional convolution kernel, the existing models can only model the changes between adjacent time points, so they still cannot achieve long-term dependence. Simply convolution after data segmentation can only obtain local information. Therefore, an improved method combining deep separable convolution with dual-head prediction is proposed, which can take into account the changes within and between cycles of data, and combine the linear and nonlinear characteristics of global data, reducing the parameters required for the convolution feature extraction process, improving prediction accuracy, and improving prediction efficiency. Summary of the invention

[0003] 1. A transformer state parameter prediction method based on deep separable convolution and dual prediction heads, characterized by the following steps:

[0004] Step 1: Assume that the original transformer parameter sequence data contains L time steps, each time step has M variables, for example, the transformer's high-voltage side load, high-voltage side load limit, oil temperature and other M indicators that affect the transformer's operating status, which are represented as a matrix And perform normalization operation X=BatchNorm(x in );

[0005] Step 2: Use the last value at the end of X Repeat filling S times;

[0006] Step 3: Divide the padded X into a block sequence with a length of P and a number of variables M according to the stride of size S. is the number of generated blocks;

[0007] Step 4: Using the Linear Projection Matrix The block sequence x p Transformed into the input sequence x d =x p W p ,in Where D is the dimension of each input sequence;

[0008] Step 5: Add xd The linear prediction head of one of the dual prediction heads is passed in, the first dimension is flattened by the Flatten function and an input dimension is passed in The output dimension is The fully connected layer gets the output Where R is the sequence length of the prediction result;

[0009] Step 6: Using the Linear Projection Matrix and position encoding Transform the sequence X into the sequence x d1 =W p1 X+W pos ,in

[0010] Step 7: Place the x d1 Pass in an input dimension The output dimension is The fully connected layer gets Where E = R + L;

[0011] Step 8: Place the x 1d The model is passed in, and the time domain data is converted to the frequency domain through fast Fourier transform, and the K frequencies with the highest amplitude in the spectrum and their corresponding period T are obtained. i and amplitude A i , where i∈{1,…,K}, the calculation formula is summarized as follows:

[0012] A=Avg(Amp(FFT(x 1d ))), {F1,…,F k} = argTopk(A),

[0013] FFT(.) and Amp(.) represent the calculation of fast Fourier transform and amplitude value. represents the calculated amplitude of each frequency, which is obtained from The average value obtained by Avg(.) in the dimension;

[0014] Step 9: Size T i The stride of the data x 1d Fill and then fill with length T i Divide the data into N i Block The formula is as follows:

[0015]

[0016] Step 10: The Inception_Block_V1 module is passed in. This module obtains features through multiple layers of depth-separable convolution. Each Inception_Block_V1 contains 6 layers of depth-separable convolution. The input and output dimensions of each convolution are consistent. The sizes of the convolution kernels are 1, 3, 5, 7, 9, and 11 respectively. Finally, the results of the 6 layers of convolution are added together and returned. The return value is passed through a nonlinear activation function GELU and then passed to the same Inception_Block_V1 module. Finally, the return value is reshaped and the result is recorded as

[0017] Step 11: Place A Add x to the weighted average 1d get The formula is:

[0018]

[0019] Step 12: Place the x dec Pass in an input dimension of The output dimension is The linear layer of , and cut off part of the result to get

[0020] Step 13: Place the x d2 As X, it passes through the same operation from step 2 to step 4 and is passed to the nonlinear prediction head, which contains an input dimension of And the output dimension is The fully connected layer and activation function GELU, finally through an input dimension of The output dimension is The fully connected layer of

[0021] Step 14: The prediction results of M different parameters that affect the transformer operation status in the next R time steps are

[0022] Step 15: Use the mean square error (MSE) loss function to calculate the loss of each channel using the following formula:

[0023]

[0024] in, represents the predicted value of the i-th time series, represents the true value of the i-th time series, summarizes the losses of each channel of the M time series, averages the summarized losses over the M time series, obtains the overall target loss, and evaluates the prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0026] Figure 1 A schematic flow chart of a transformer power load and oil temperature prediction method using deep separable convolution and dual-head prediction provided in an embodiment of the present invention;

[0027] Figure 2 This is the internal flow structure diagram of the nonlinear prediction head;

[0028] Figure 3 This is the internal process structure diagram for feature extraction;

[0029] Figure 4 It is a schematic diagram of the structure of the model based on the architecture of CNN feature extraction module;

[0030] Figure 5 To compare with the mainstream time series prediction models based on CNN time series prediction models and other architectures; DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings;

[0032] like Figure 1 As shown, an embodiment of the present invention provides a transformer state parameter prediction method based on deep separable convolution and dual prediction heads, which is characterized by comprising the following steps:

[0033] Step 1: Collect the original transformer parameter sequence data of a certain place, which contains 96 time steps. Each time step has 7 variables, namely, high voltage side load, high voltage side load limit, medium voltage side load, medium voltage side load limit, low voltage side load, low voltage side load limit, and oil temperature, which are represented as a matrix And perform normalization operation X=BatchNorm(x in );

[0034] Step 2: Use the last value at the end of X Perform 8 repeated fillings;

[0035] Step 3: Divide the padded X into a block sequence with a length of 16 and a variable number of 7 according to a stride of 8. Among them, 12 is the number of blocks calculated;

[0036] Step 4: Using the Linear Projection Matrix The block sequence x p Transformed into the input sequence x d =x p W p ,in Where 512 is the dimension of each input sequence;

[0037] Step 5: Add x d The linear prediction head of one of the dual prediction heads is passed in, the first dimension is flattened by the Flatten function and an input dimension is passed in The output dimension is The fully connected layer gets the output Among them, 96 is the sequence length of the prediction result;

[0038] Step 6: Using the Linear Projection Matrix and position encoding Transform the sequence X into the sequence x d1 =W p1 X+W pos ,in

[0039] Step 7: Place the x d1 Pass in an input dimension The output dimension is The fully connected layer gets

[0040] Step 8: Place the x 1d The model is passed in, and the time domain data is converted to the frequency domain through fast Fourier transform, and the five frequencies with the highest amplitude in the spectrum and their corresponding periods T are obtained. i and amplitude A i , where i∈{1,…,5}, the calculation formula is summarized as follows:

[0041] A=Avg(Amp(FFT(x 1d ))),{F1,…,F k}=argTopk(A),

[0042] FFT(.0 and Amp(.) represent the calculation of fast Fourier transform and amplitude value. represents the calculated amplitude of each frequency, which is obtained from The average value obtained by Avg(.) in the dimension;

[0043] Step 9: Size T i The stride of the data x 1dFill and then fill with length T i Divide the data into N i Block A total of 5 x 2d ;

[0044] The formula is as follows:

[0045]

[0046] Step 10: Place each The Inception_Block_V1 module is passed in. This module extracts features through multiple layers of depth-wise separable convolution. Each Inception_Block_V1 contains 6 layers of depth-wise separable convolution. The input and output dimensions of each convolution are consistent. The sizes of the convolution kernels are 1, 3, 5, 7, 9, and 11 respectively. Finally, the results of the 6 layers of convolution are added together and returned. The return value is passed through a GELU activation function and then passed to the same Inception_Block_V1 layer. Finally, the return value is reshaped and the result is recorded as

[0047] Step 11: Place A Add x to the weighted average 1d get The formula is:

[0048]

[0049] Step 12: Place the x dec Pass in an input dimension of The output dimension is The linear layer of , and cut off part of the result to get

[0050] Step 13: Place the x d2 As X, it passes through the same operation from step 2 to step 4 and is passed to the nonlinear prediction head, which contains an input dimension of And the output dimension is The fully connected layer and activation function GELU, finally through an input dimension of The output dimension is The fully connected layer of

[0051] Step 14: The prediction results of 7 different parameters affecting the transformer operation status in the next 96 time steps are

[0052] Step 15: Use the mean square error (MSE) loss function to calculate the loss of each channel using the following formula:

[0053]

[0054] in, represents the predicted value of the i-th time series, Represents the true value of the i-th time series, 7 represents the number of time series, the losses of each channel of the 7 time series are summarized, the summarized losses are averaged over the 7 time series to obtain the overall target loss, and the prediction results are evaluated.

[0055] like Figure 5 As shown in the figure, this model (named DH_Net) is compared with 3 other current mainstream models based on CNN architecture, 1 based on RNN architecture, and 1 based on Transformer architecture. The results are model performance evaluation based on the public ETTh2 dataset, which is a time series of 7 parameters affecting the operating status of transformers collected every 1 hour from January 2016 to June 2018. Input Length and Output Length are the input sequence length and prediction sequence length of the model, respectively. MSE and MAE are the mean square error and mean absolute error calculated by comparing the predicted value with the true value. The comparison results show that this model has a smaller prediction error than other current mainstream models for this dataset when the input sequence length is 96 and the prediction length is 96, 192, 336, and 720, respectively, which means that the prediction is more accurate.

[0056] Finally, the method of the present invention can be used to detect potential faults or abnormalities of the transformer (such as overheating, overload, etc.) at an early stage, and can help operation and maintenance personnel take measures to maintain or replace equipment in a timely manner to avoid sudden faults causing power outages, equipment damage, casualties and other safety risks or unnecessary economic losses.

Claims

1. A transformer state parameter prediction method based on deep separable convolution and dual prediction heads, characterized by the following steps: Step 1: Assume that the original transformer parameter sequence data contains L time steps, each time step has M variables, for example, the transformer's high-voltage side load, high-voltage side load limit, oil temperature and other M indicators that affect the transformer's operating status, which are represented as a matrix And perform normalization operation X = BatchNorm (x in ); Step 2: Use the last value at the end of X Repeat filling S times; Step 3: Divide the padded X into a block sequence with a length of P and a number of variables M according to the stride of size S. is the number of generated blocks; Step 4: Using the Linear Projection Matrix The block sequence x p Transformed into input sequence x d =x p W p ,in Where D is the dimension of each input sequence; Step 5: Add x d The linear prediction head of one of the dual prediction heads is passed in, the first dimension is flattened by the Flatten function and an input dimension is passed in The output dimension is The fully connected layer gets the output Where R is the sequence length of the prediction result; Step 6: Using the Linear Projection Matrix and position encoding Transform the sequence X into the sequence x d1 =W p1 X+W pos ,in Step 7: Place the x d1 Pass in an input dimension The output dimension is The fully connected layer gets Where E = R + L; Step 8: Place the x 1d The model is passed in, and the time domain data is converted to the frequency domain through fast Fourier transform, and the K frequencies with the highest amplitude in the spectrum and their corresponding period T are obtained. i and amplitude A i , where i∈{1,…,K}, the calculation formula is summarized as follows: A=Avg(Amp(FFT(x 1d ))), {F1,…,F k } = argTopk(A), FFT(.) and Amp(.) represent the calculation of fast Fourier transform and amplitude value. represents the calculated amplitude of each frequency, which is obtained from The average value obtained by Avg(.) in the dimension; Step 9: Size T i The stride of the data x 1d Fill and then fill with length T i Divide the data into N i Block The formula is as follows: Step 10: The Inception_Block_V1 module is passed in, which extracts features through multiple layers of depth-separable convolution. Each Inception_Block_V1 contains 6 layers of depth-separable convolution. The input and output dimensions of each convolution are consistent. The sizes of the convolution kernels are 1, 3, 5, 7, 9, and 11 respectively. Finally, the results of the 6 layers of convolution are added together and returned. The return value is passed through a nonlinear activation function GELU and then passed to the same Inception_Block_V1 module. Finally, the return value is reshaped and the result is recorded as Step 11: Place A Add x to the weighted average 1d get The formula is: Step 12: Place the x dec Pass in an input dimension of The output dimension is The linear layer of , and cut off part of the result to get Step 13: Place the x d2 As X, it passes through the same operation from step 2 to step 4 and is passed to the nonlinear prediction head, which contains an input dimension of And the output dimension is The fully connected layer and activation function GELU, finally through an input dimension of The output dimension is The fully connected layer of Step 14: The prediction results of M different parameters that affect the transformer operation status in the next R time steps are Step 15: Use the mean square error (MSE) loss function to calculate the loss of each channel using the following formula: in, represents the predicted value of the i-th time series, represents the true value of the i-th time series, summarizes the losses of each channel of the M time series, averages the summarized losses over the M time series, obtains the overall target loss, and evaluates the prediction results.

Images