An integrated energy system bad data preprocessing method and system

Abstract

The application discloses a kind of comprehensive energy system bad data preprocessing method and system, belong to comprehensive energy system field;A kind of comprehensive energy system bad data preprocessing method includes: first, the real-time measurement data of comprehensive energy system state is collected, and normalization processing is carried out;Then input comprehensive energy system bad data detection model, if detection model determines that there is bad data in measurement data, then identification and correction are carried out;If detection model determines that measurement data is normal measurement data, then end data preprocessing procedure;Finally, the position of bad data is determined using comprehensive energy system bad data identification correction model and it is corrected, and end data preprocessing procedure;The method can detect out random error too big, measurement loss and malicious data attack etc. bad data in real time, accurately according to the reconstruction error of measurement data, compared with traditional supervised learning algorithm, it has more accurate detection effect when training sample is less.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of integrated energy systems, and particularly relates to an integrated energy system bad data preprocessing method and system. BACKGROUND

[0002] Accurate and real-time acquisition of the operation state of an integrated energy system (IES) is a prerequisite for its optimal scheduling; however, the data directly obtained by the existing measurement system cannot accurately and comprehensively describe the operation state of the IES, and the reasons include: 1) the number of measurement devices is limited, and it is difficult to cover all nodes; 2) due to factors such as device precision and human operation, the measurement information always has certain errors. Therefore, state estimation is needed to obtain the real-time and accurate operation state of the IES based on imperfect and inaccurate measurement information. At the same time, sensor failures, communication line interferences, and environmental deterioration in the IES can all cause the errors of the measurement data to exceed the specified range, become abnormal data, and even cause the loss of the measurement data. Therefore, it is necessary to consider detecting abnormal data and repairing abnormal data by using a data preprocessing method before state estimation, which can not only improve the accuracy of the measurement data, but also provide reliable data support for subsequent state estimation work.

[0003] The current IES state estimation mainly adopts the traditional model-based bad data preprocessing method, which is relatively stable in overall effect, but needs to be calculated multiple times, has low calculation efficiency, and since the preprocessing step is performed after state estimation, residual error flooding and residual error pollution are prone to occur; therefore, in order to improve the robustness of the IES state estimation and avoid damage to the safe operation of the IES caused by various bad data, it is necessary to explore a bad data preprocessing method based on data driving and suitable for massive monitoring data of the IES. SUMMARY

[0004] In view of the deficiencies of the prior art, the purpose of the present application is to provide an integrated energy system bad data preprocessing method and system.

[0005] The purpose of the present application can be achieved by the following technical solutions:

[0006] An integrated energy system bad data preprocessing method comprises the following steps:

[0007] Collecting real-time measurement data of the state of the integrated energy system and performing normalization processing;

[0008] The normalized real-time measurement data is input into the comprehensive energy system bad data detection model, if the detection model determines that there is bad data in the measurement data, identification and correction are performed, if the detection model determines that the measurement data is normal measurement data, the data preprocessing process is ended.

[0009] The position of the bad data is determined by using the comprehensive energy system bad data identification and correction model, and the bad data is corrected, and the data preprocessing process is ended.

[0010] Further, the comprehensive energy system measurement data types include: grid node voltage amplitude, grid branch active power, grid branch reactive power, grid node active power, grid node reactive power, heat network node heat power, heat network node pressure, heat network pipeline flow, heat network node water supply temperature, heat network node return water temperature, gas network pipeline flow, gas network node pressure and gas network node injection flow.

[0011] Further, the comprehensive energy system bad data detection model is composed of a plurality of AE bad data detection models, and the specific steps of using the comprehensive energy system bad data detection model to perform data determination work include:

[0012] S21, the historical measurement data of the comprehensive energy system is normalized, superimposed with Gaussian white noise, and divided into a training set, a validation set and a test set;

[0013] S22, the historical measurement data of the comprehensive energy system is divided into a plurality of sub-data sets according to categories, and an AE bad data detection model is built for each type of measurement data, taking reconstruction error as the loss function, and the parameters of each type of bad data detection model are optimized through multiple training;

[0014] S23, the validation set data is input into the trained AE bad data detection model, the minimum reconstruction error of the bad data is calculated and determined, and is used as the bad data detection threshold; if the reconstruction error of a certain data is greater than the detection threshold, it is regarded as bad data, otherwise it is normal data; the bad data threshold is set from low to high in turn, the F1 score of the validation set under different thresholds is calculated, and the threshold with the highest score is selected as the online detection threshold of the bad data;

[0015] S24, the normalized real-time measurement data is input into the corresponding trained AE bad data detection model according to the category, the reconstruction error of each type of measurement data at the current time section is calculated, and compared with the bad data detection threshold of the corresponding model, if it is greater than the detection threshold, it is determined that there is bad data in the real-time measurement data, otherwise it is determined to be normal measurement.

[0016] Further, the AE includes an encoder and a decoder, and the calculation process of the encoder is:

[0017] X = f encode (I) = σ encode (W encode I + b encode )

[0018] where I is the input vector of the autoencoder; f encode is the encoder function; X is the encoded information vector of the input; W enocde is the parameter vector of the encoder neurons; b encode is the bias vector of the encoder neurons; σ encode is the activation function of the encoder neurons;

[0019] The calculation process of the decoder is as follows:

[0020] O = f decode (X) = σ decode (W decode X + b decode )

[0021] where O is the output vector of the autoencoder; f decode is the decoder function; W deocde is the parameter vector of the decoder neurons; b decode is the bias vector of the decoder neurons; σ decode is the activation function of the decoder neurons;

[0022] The autoencoder continuously updates the neuron parameters through training to minimize the reconstruction error of the input I and the output O, and the calculation process of the reconstruction error is as follows:

[0023]

[0024] where e autoencoder is the reconstruction error of the autoencoder; J is the reconstruction loss function of the autoencoder, usually the mean square error loss function; n is the dimension of the input vector; I i and O i are the i-th components of the input vector and the output vector, respectively.

[0025] Further, the calculation formula of the F1 score is as follows:

[0026]

[0027]

[0028]

[0029] where S precision , S recall , S F1Precision, recall and F1-score of the sample, respectively; n TP FP TN FN TP, FP, TN and FN in the sample, respectively; TP is a correct positive example, FP is a false positive example, TN is a correct negative example, FN is a false negative example, and represents a correct sample being predicted as false; wherein the bad data is regarded as a positive example and the normal data is regarded as a negative example.

[0030] Further, the comprehensive energy system bad data identification correction model is composed of a plurality of GRU-Attention bad data identification correction models, and the steps of identifying and correcting the bad data using the comprehensive energy system bad data identification correction model specifically include:

[0031] S31, normalizing the comprehensive energy system historical measurement data, superimposing Gaussian white noise, and dividing the training set, the validation set and the test set;

[0032] S32, dividing the comprehensive energy system historical measurement data into a plurality of sub-data sets according to categories, respectively building various GRU-Attention bad data identification correction models for each type of measurement data; and setting hyperparameters to train the models multiple times to optimize the model parameters;

[0033] S33, inputting the validation set data into the trained GRU-Attention bad data identification correction model, recording the maximum relative error of the prediction of the validation set, and taking it as the online identification threshold of the bad data;

[0034] S34, reading the historical data of a certain type of measurement data with bad data in a time period, performing normalization processing, and inputting the trained GRU-Attention bad data identification correction model, comparing the predicted values of the measurement data of each measuring point with the real-time measurement data, and if the relative error is greater than the identification threshold, determining that the measuring point is the position of the bad data; for the position of the bad data, the measurement data prediction value is used to replace it.

[0035] Further, the GRU-Attention bad data identification correction model includes a gate recurrent unit network (Gate Recurrent Unit, GRU) and an attention mechanism (Attention); the relationship inside the GRU is:

[0036] z t =σ(W z ·[h t-1 ,x t ])

[0037] r​​​t = σ(W r · [h t-1 , x t ])

[0038] h t ' = tanh(W h · [r t *h t-1 , x t ])

[0039] h t = (1 - z t )*h t-1 + z t *h t '

[0040] wherein z t is the update gate of the GRU unit; r t is the reset gate of the GRU unit; x t is the input signal of the GRU unit at the current time; h t-1 , h t are the outputs of the GRU unit at the previous time and the current time, respectively; W z , W r , W h represent the weight coefficients of each neuron of the GRU unit; σ is an activation function; tanh is a hyperbolic tangent activation function.

[0041] Further, the basic mathematical expression of the Attention mechanism is:

[0042] s i = f score (u, v i )

[0043]

[0044] c = a i v i

[0045] f score (u, v i ) = u T W s v i

[0046] wherein u is the feature vector at the current time step; v i is the feature vector with the timestamp i in the time series data; s i is the attention score; f score is the attention calculation function; a i is the normalized attention score; c is the feature sequence vi a feature of the calculated current timestamp; W s network parameters for the attention mechanism.

[0047] Further, only normal measurement data is in the training set, and the verification set and the test set are composed of normal measurement data and bad data, and the construction method and the composition ratio of the bad data in the verification set and the test set are consistent.

[0048] A bad data preprocessing system of an integrated energy system, comprising:

[0049] A data acquisition module: acquiring real-time measurement data of the integrated energy system state and performing normalization processing;

[0050] A bad data detection module: inputting the normalized real-time measurement data into an integrated energy system bad data detection model, if the detection model determines that there is bad data in the measurement data, identifying and correcting it; if the detection model determines that the measurement data is normal measurement data, ending the data preprocessing process;

[0051] And a bad data identification and correction module: determining the position of the bad data by using an integrated energy system bad data identification and correction model and correcting it, and ending the data preprocessing process.

[0052] Advantages of the present application:

[0053] 1. The IES bad data detection model based on AE established by the present application can detect random error too large, measurement loss and malicious data attack and other bad data in real time and accurately according to the reconstruction error of the measurement data; compared with the traditional supervised learning algorithm, it has more accurate detection effect when the training sample is less.

[0054] 2. The IES bad data identification and correction model based on GRU-Attention established by the present application can more accurately predict the measurement compared with the traditional model-based algorithm and RNN, LSTM and other intelligent algorithms, and the use of the Attention mechanism further enhances the prediction effect of the model; by comparing the difference between the real measurement and the predicted value at the current time, the position of each bad data can be accurately located.

[0055] 3. In the bad data preprocessing method designed by the present application, only when the bad data detection model determines that there is bad data at the current measurement point, the bad data identification and correction model will start, which has good data preprocessing calculation efficiency and can effectively process the massive monitoring data of the integrated energy system.

[0056] 4. The bad data preprocessing method designed by the application is matched with the subsequent state estimation algorithm, bad data in real-time measurement can be effectively detected and identified, and the bad data is corrected, the robustness of IES state estimation is further improved, and the bad data detection and identification model only needs to be detected once for each measurement vector, the time complexity is low, and the real-time performance of the IES state estimation algorithm can be effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0058] Figure 1 is the method implementation flowchart of the application;

[0059] Figure 2 is the system framework structure of the application;

[0060] Figure 3 is the auto-encoder structure diagram of the application;

[0061] Figure 4 is the GRU-Attention bad data identification and correction model structure diagram of the application;

[0062] Figure 5 is the electric-gas-heat integrated energy system topology diagram of the application;

[0063] Figure 6 is the prediction relative error comparison diagram of part of the measurement data of the application;

[0064] Figure 7 is the state estimation maximum relative error diagram of each method under scenario 5 of the application;

[0065] Figure 8 is the state estimation maximum relative error diagram of each method under scenario 6 of the application;

[0066] Figure 9 is the state estimation maximum relative error diagram of each method under scenario 7 of the application. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0068] As shown in Figure 1 and Figure 2 A comprehensive energy system bad data preprocessing method, comprising the following steps:

[0069] S1, collecting IES state real-time measurement data through a measurement acquisition device and performing normalization processing;

[0070] Wherein, the IES measurement data types include: grid node voltage amplitude, grid branch active power, grid branch reactive power, grid node active power, grid node reactive power, heat network node heat power, heat network node pressure, heat network pipeline flow, heat network node water supply temperature, heat network node return water temperature, gas network pipeline flow, gas network node pressure and gas network node injection flow. Different types of measurement data can be collected by corresponding measurement acquisition devices.

[0071] Considering that the numerical value range and amplitude of each measurement in the IES are inconsistent, in order to avoid the negative impact of dimension difference on artificial intelligence model, Min-Max normalization is used to normalize each type of measurement data involved, and the calculation method is as follows:

[0072]

[0073] In the formula, u is the normalized input vector; u i is the i-th component of the vector u; u' i is the i-th component of the vector u after normalization; min(u), max(u) represent the minimum component and the maximum component of the vector u, respectively.

[0074] S2, input the normalized real-time measurement data into the comprehensive energy system bad data detection model, if the detection model determines that there is bad data in the measurement data, then go to S3 for further identification and correction; if the detection model determines that the measurement data is normal measurement data, directly input the real-time measurement data into the subsequent state estimation model, and end the current data preprocessing process;

[0075] Wherein, the comprehensive energy system bad data detection model is composed of multiple AE bad data detection models, and the specific steps of using the comprehensive energy system bad data detection model for determination include:

[0076] S21, normalizing the IES historical measurement data, superimposing Gaussian white noise, dividing it into training set, validation set and test set, the training set only contains normal measurement data, the validation set and the test set are composed of normal measurement data and bad data, and the construction method and the composition ratio of the bad data in the validation set and the test set are consistent.

[0077] S22, the IES historical measurement data is divided into multiple sub-data sets according to categories, and an AE (autoencoder) bad data detection model is built for each type of measurement data to optimize the parameters of each type of bad data detection model through multiple training by taking reconstruction error as the loss function;

[0078] AE (autoencoder) is a branch of artificial neural network, commonly used in semi-supervised learning tasks and unsupervised learning tasks, which can take input information as learning target, perform representation learning on input information, and reconstruct input data through neural network to make the output of the model equal to the input; AE takes minimizing the error between input and output as the goal, and iteratively optimizes the neuron parameters through gradient descent, thereby obtaining different representation methods of input data, and completing the functions of data denoising, dimensionality reduction and anomaly detection.

[0079] As shown in Figure 3 , the autoencoder is usually composed of an encoder and a decoder, the function of the encoder is to encode the input of the autoencoder and compress it into new representation information; and the function of the decoder is to reconstruct the encoded information to make the representation information return to the original input; the calculation process of the encoder is:

[0080] X=f encode (I)=σ encode (W encode I+b encode )

[0081] In the formula, I is the input vector of the autoencoder; f encode is the encoder function; X is the information vector after the input is encoded; W enocde is the parameter vector of the encoder neuron; b encode is the bias vector of the encoder neuron; σ encode is the activation function of the encoder neuron;

[0082] The calculation process of the decoder is:

[0083] O=f decode (X)=σ decode (W decode X+b decode )

[0084] In the formula, O is the output vector of the autoencoder; f decode is the decoder function; W deocde is the parameter vector of the decoder neuron; b decode is the bias vector of the decoder neuron; σ decode is the activation function of the decoder neuron;

[0085] An autoencoder continuously updates neuron parameters during training to minimize the reconstruction error between input I and output O. The reconstruction error is calculated as follows:

[0086]

[0087] In the formula, e autoencoder Let be the reconstruction error of the autoencoder; J be the reconstruction loss function of the autoencoder, usually the mean squared error loss function; n be the dimension of the input vector; I i and O i These are the i-th components of the input vector and the output vector, respectively;

[0088] In anomaly detection, autoencoders output reconstructed information and calculate the reconstruction error between input and output by unsupervised learning of normal input data. After the encoding and decoding operations of the autoencoder, the reconstruction error of abnormal data is larger than that of normal data, thus realizing the detection of abnormal data. In addition to the classic autoencoder, there are many autoencoder variants for different scenarios, such as sparse autoencoders, convolutional autoencoders, regularized autoencoders, and denoising autoencoders, etc.

[0089] S23. Input the validation set data into the trained AE bad data detection model, calculate and determine the minimum reconstruction error of bad data, and use it as the bad data detection threshold. If the reconstruction error of a data is greater than the detection threshold, it is considered bad data; otherwise, it is normal data. Set the bad data thresholds from low to high, calculate the F1 score of the validation set at different thresholds, and select the case with the highest score as the online detection threshold for bad data.

[0090] The F1 score is the harmonic mean of precision and recall; a higher F1 score indicates more reliable predictions. The F1 score can be calculated using the following formula:

[0091]

[0092]

[0093]

[0094] In the formula, S precision S recall S F1 These represent the precision, recall, and F1 score of the sample, respectively; n TP n FP n TN n FNTP, FP, TN, FN represent the number of TP, FP, TN, FN in the sample respectively; TP is a correct positive example, which represents that a correct sample is predicted correctly; FP is a false positive example, which represents that an incorrect sample is predicted correctly; TN is a correct negative example, which represents that an incorrect sample is predicted incorrectly; FN is a false negative example, which represents that a correct sample is predicted incorrectly; wherein, the abnormal data is regarded as a positive example, and the normal data is regarded as a negative example.

[0095] S24, input the normalized real-time measurement data into the corresponding trained AE abnormal data detection model according to the category, calculate the reconstruction error of each category of measurement data at the current time section, and compare it with the abnormal data detection threshold of the corresponding model, if it is greater than the detection threshold, it is determined that there is abnormal data in the real-time measurement data of this category, otherwise it is determined to be normal measurement.

[0096] S3, using the comprehensive energy system abnormal data identification correction model, determining the location of the abnormal data and correcting it, inputting the corrected measurement data into the subsequent state estimation model, and ending the current data preprocessing process;

[0097] The comprehensive energy system abnormal data identification correction model is composed of multiple GRU-Attention abnormal data identification correction models, and the steps of using the comprehensive energy system abnormal data identification correction model to identify and correct the abnormal data include:

[0098] S31, normalizing the IES historical measurement data, superimposing Gaussian white noise, and dividing the training set, the validation set and the test set, the training set only has normal measurement data, and the validation set and the test set are composed of normal measurement data and abnormal data;

[0099] Set the size of the sliding window t step The measurement data of each category from time t-1 to time t-t step -1 is used as the input of each model, and the output of each model is the predicted value of each category of measurement data at time t;

[0100] S32, divide the IES historical measurement data into multiple sub-data sets according to the category, build each GRU-Attention abnormal data identification correction model for each category of measurement data respectively; and by setting the hyperparameters of the GRU-Attention identification correction model, the model is trained multiple times, taking MSE as the loss function, and the model parameters are updated and optimized constantly;

[0101] GRU is a variant of LSTM, and LSTM is also proposed to solve the long-term dependence and gradient explosion problem of traditional RNN; GRU uses reset gate and update gate to control its input and output, and the network structure is simpler than RNN and LSTM; The internal relationship of GRU network is as follows:

[0102] z t =σ(W z ·[h t-1 ,x t ])

[0103] r t =σ(W r ·[h t-1 ,x t ])

[0104] h′ t =tanh(W h ·[r t *h t-1 ,x t ])

[0105] h t =(1-z t )*h t-1 +z t *h′ t

[0106] In the formula, z t is the update gate of the GRU unit, which is used to control how much information of the hidden layer output at the previous moment needs to be forgotten; r t is the update gate of the GRU unit, which is used to control how much historical information needs to be retained; x t is the input signal of the GRU unit at the current moment; h t-1 , h t are the outputs of the GRU unit at the previous moment and the current moment respectively; W z , W r , W h respectively represent the weight coefficients of each neuron of the GRU unit; σ is the activation function, usually the sigmoid activation function; tanh is the hyperbolic tangent activation function.

[0107] The representation of the sigmoid activation function and the tanh activation function is as follows:

[0108]

[0109]

[0110] In the formula, f sigmoid , f tanhrespectively represent sigmoid activation function and tanh activation function; x represents the input of the activation function.

[0111] Attention mechanism originates from the study of human visual characteristics; in the process of observing information, humans usually selectively focus on key information and ignore other unimportant information. This mechanism is called Attention mechanism. The basic mathematical expression of Attention is:

[0112] s i =f score (u,v i )

[0113]

[0114] c=α i v i

[0115] f score (u,v i )=u T W s v i

[0116] In the formula, u is the feature vector in the current time step; v i is the feature vector with timestamp i in the time series data; s i is the attention score; f score is the attention calculation function; α i is the normalized attention score; c is the feature of the current timestamp calculated based on the attention score and the feature sequence v i ; W s is the network parameter of the attention mechanism, obtained by training.

[0117] The overall structure of the GRU-Attention bad data identification correction model is shown in Figure 4 The GRU neural network includes two layers, one layer is used to learn the measurement time series data, and the second layer is learned in reverse based on the output of the first layer; then the output of the GRU is learned using the Attention mechanism, the influence degree of each node output of the GRU on the measurement prediction value is calculated, and the relationship between the GRU output and the measurement prediction value is established.

[0118] S33, input the verification set data into the trained GRU-Attention bad data identification correction model, record the maximum relative error of the verification set, and take it as the online identification threshold of the bad data;

[0119] S34, read the measurement data of a certain type of bad data existing at time t-1 to time t-tstep The historical data of 1 is normalized, and is input into the trained GRU-Attention bad data identification correction model, the predicted values of the measurement data of each measuring point output by the model are compared with the real-time measurement data, if the relative error is greater than the identification threshold, it is determined that the measuring point is the position where the bad data occurs; for the position where the bad data occurs, the measurement data prediction value is used to replace it, and the correction result is returned for subsequent comprehensive energy system state estimation. Embodiments

[0121] Based on the above scheme, the above method is applied as an example, and the specific implementation is as follows:

[0122] An IEEE14-node power system, a 15-node regional heat system, a 14-node regional heat system, and two cogeneration units constitute an electricity-gas-heat comprehensive energy system as an example for simulation analysis. The topological structure of the network is shown in Figure 5 The load curve is sampled once every 1 hour, and there are a total of 8760 data in a year. Based on the multi-energy flow calculation method of the comprehensive energy system, the operation state of the example is simulated to obtain 8760 power flow true values. The power flow true values are used as the data here. The measurement random error of the power flow true values is simulated by adding Gaussian white noise with a standard deviation of 0.005 to the measurement, and the power flow data is divided into a training set, a validation set, and a test set in a ratio of 8:1:1. The training set has a total of 7008 data, the validation set has a total of 876 data, and the test set has a total of 876 data.

[0123] Considering the differences in informatization and automation construction of each energy subsystem, real-time measurement devices are configured for all nodes and branches of the power system; measurement devices are configured for the heat system and the natural gas system according to the principle of preferential trunk, as shown in Table 1:

[0124] Table 1 Measurement configuration of comprehensive energy system

[0125]

[0126] The effect evaluation index used here is the relative error, the average relative error, and the maximum relative error, and the formula is:

[0127]

[0128]

[0129]

[0130] In the formula, e r , e ave , and e max are the relative error, the average relative error, and the maximum relative error, respectively; xi,true , x i are the true value and the estimated value of the state variable respectively; n is the sample number.

[0131] The proportion of three types of bad data, random error, measurement loss, malicious attack and normal measurement in the training set is 0.05:0.05:0.05:0.85. The setting of the three types of bad data is as follows:

[0132] ① The maximum number of random errors under the same section is n r max = 10, and the error interval of the random error is [0.5, 0.8].

[0133] ② The maximum number of measurement losses under the same time section is 5.

[0134] ③ The size of the malicious attack is n a = 15, and the intensity A of the malicious attack is 0.2.

[0135] Case result one: effect analysis of the bad data detection model of the integrated energy system

[0136] After multiple training and verification, the epoch of the self-encoder during training is 100, the batch size is 32, the adopted optimizer is Adam, and the learning rate is 0.1. In order to quantitatively analyze the effectiveness of the bad data detection model proposed in the application, the bad data under multiple scenes are tested respectively, and the test results are compared with other bad data detection methods, including random forest and support vector machine (SVM). The bad data scenes set include:

[0137] Scene 1: 30% of the normal data in the training set is replaced by bad data with too large random error.

[0138] Scene 2: 30% of the normal data in the test set is replaced by bad data of measurement loss.

[0139] Scene 3: 30% of the normal data in the test set is replaced by bad data of malicious attack.

[0140] Scene 4: 10% of the normal data in the test set is replaced by bad data with too large random error; 10% of the normal data is replaced by bad data of measurement loss; and 10% of the normal data is replaced by bad data of malicious attack.

[0141] The bad data detection result under scenario 1 is shown in Table 2. It can be seen from the table that for the random error type bad data, the precision and recall of the random forest are close, 0.9574 and 0.9438 respectively, the recall of the SVM is lower than that of the random forest, the recall of the application is slightly higher than that of the random forest, but the precision of the SVM and the application is 1. The case shows that for the random error type bad data, the SVM and the application can accurately identify normal measurement, and there is no false detection, but the relatively low recall shows that both of them have missed detection. From the last F1 score, it can be seen that the comprehensive effect of the application is better than that of the random forest and the SVM.

[0142] Table 2 Bad data detection result under scenario 1

[0143]

[0144] The bad data detection result under scenario 2 is shown in Table 3. It can be seen from the table that the precision and recall of the random forest, the SVM and the application are obviously improved compared with scenario 1. The case shows that compared with the random error type bad data, the measurement loss type bad data is easier to identify, and the recall of the application reaches 0.97, which is at a high level. Under the premise of no false detection, the application can better detect the measurement loss type bad data.

[0145] Table 3 Bad data detection result under scenario 2

[0146]

[0147] The bad data detection result under scenario 3 is shown in Table 4. It can be seen from the table that the precision and recall of the three types of algorithms for scenario 3 are lower than those of scenario 1 and scenario 2, which shows that the detection difficulty of the malicious data attack set by the application is greater than that of the random error and the measurement loss. The precision of the application and the SVM is not 1, which shows that both of them have the possibility of false detection for the malicious data attack type bad data. But the F1 score of the application is still greater than that of the random forest and the SVM, which shows that the malicious data attack detection effect of the application is better than that of the two.

[0148] Table 4 Bad data detection result under scenario 3

[0149]

[0150] The bad data detection results under scenario 4 are shown in Table 5. As can be seen from the table, if three types of bad data are mixed in the test set at the same time, the precision and recall of the random forest are 0.9614 and 0.9458 respectively, the false detection and missed detection probabilities are relatively close, and the precision and recall of the SVM are 0.9908 and 0.9194 respectively, the false detection probability is relatively low, but the missed detection probability is relatively high. The precision and recall of the present application are higher than those of the random forest and the SVM, so that the F1 score of the present application leads the random forest and the SVM, reaching 0.9732. In summary, the comprehensive effect of the bad data detection of the present application is the best among the three methods.

[0151] Table 5 Bad data detection results under scenario 4

[0152]

[0153] Case result two: effect analysis of bad data identification correction model of integrated energy system

[0154] After multiple training and parameter adjustment, the time step of the constructed GRU is 24, the optimizer is Adam, the learning rate is 0.03, the epoch is 300, the batch size is 32, the simple Attention mechanism is adopted, and the number of hidden layers of the GRU is 2. To verify the effectiveness of the bad data identification correction model of the present application, the model is compared with other similar algorithms, including RNN, LSTM and GRU without considering the Attention mechanism. After screening of the validation set, the selected bad data correction threshold of the present application is 4.87%, and the bad data correction thresholds of RNN, LSTM and GRU are 9.62%, 6.18% and 5.44% respectively.

[0155] The prediction average relative error and maximum relative error of the bad data identification correction model in the test set are shown in Table 6. As can be seen from the table, when the average relative error and maximum relative error of the present application algorithm are the lowest, the overall effect is the best. Due to its own defects, the prediction effect of RNN is the worst among the four methods, and the maximum relative error reaches 9.63%, which is difficult to meet the requirements of prediction measurement. LSTM and GRU are improved algorithms proposed for the shortcomings of RNN, and their correction effects are obviously better than that of RNN, and the average relative error is far below the error range of bad data. Compared with LSTM, GRU has a simpler network structure, and in the case of smaller training data, the correction effect is better, and the average relative error and maximum relative error are reduced by 20.08% and 11.51% respectively. Since the present application adds the Attention mechanism to the GRU, the mapping relationship between the GRU output and the predicted value of the measurement is more reasonable, so the average relative error and maximum relative error of the present application are further reduced compared with the GRU.

[0156] Table 6 Comparison of the effect of measurement prediction

[0157]

[0158] To further analyze the effectiveness of the present application, 50 time section bad data are randomly selected from the test set, and the bad data recognition correction model is used to predict them, and the relative error is compared with the test results of other algorithms, and part of the comparison results of the measurement are as shown in Figure 6 Figure 6 (a) in the (a) is the active power of the grid branch, Figure 6 (b) in the (b) is the grid node voltage amplitude, Figure 6 (c) in the (c) is the heat network node heat power, Figure 6 (d) in the (d) is the heat network pipeline flow, Figure 6 (e) in the (e) is the gas network pipeline flow, Figure 6 (f) in the (f) is the gas network node pressure.

[0159] Case result three: effect analysis of comprehensive energy system bad data preprocessing method

[0160] To verify the effectiveness of the present application, it is combined with the comprehensive energy system state estimation model to analyze the robustness improvement effect of the deep learning based comprehensive energy system state estimation model. The CNN comprehensive energy system state estimation model is used, and the bad data are mixed in the training set. The proportion of normal data, random error type bad data, measurement loss type bad data and malicious attack type bad data in the training set is 85%, 5%, 5% and 5% respectively. To quantitatively analyze the robustness improvement effect, three bad data scenarios are set to test the combination of bad data recognition correction model and CNN state estimation model. The three bad data scenarios are as follows:

[0161] Scenario 5: modify the test set to random error type bad data.

[0162] Scenario 6: modify the test set to measurement loss type bad data.

[0163] Scenario 7: modify the test set to malicious attack type bad data.

[0164] WLS (with bad data recognition link), WLAV, CNN, and CNN with the present application method are respectively tested for state estimation effect, and the maximum relative error of the four methods in the test is as shown in Figure 7 , Figure 8 and Figure 9 ​It can be seen from the above table that after the data preprocessing of the application is used for the CNN, the maximum relative error of state estimation is significantly reduced, and the overall effect is better than WLS, WLAV and CNN. In scenario 3, the maximum relative error of state estimation after using the bad data detection model is the largest, much larger than the reduction in scenarios 1 and 2, indicating that the robustness of the application to malicious attack type bad data is most obvious. This situation shows that the application can accurately detect and correct bad data such as random errors, measurement loss and malicious attacks before entering the deep learning state estimation model, thereby minimizing the impact of bad data on the state estimation model. The bad data identification and correction model of the application is most effective for malicious attack type bad data, because the application is constructed based on the data-driven idea, and malicious attack type bad data cannot be constructed against the bad data identification and correction model, so that malicious attacks and other bad data are as easy to be identified by the application. Using the application for deep learning type comprehensive energy system state estimation model can effectively reduce the influence of bad data on state estimation results, and the application makes the deep learning state estimation model not need to be supervised by the bad data, thereby avoiding the problem of precision decline caused by overfitting of the state estimation model.

[0165] Case result four: analysis of the calculation efficiency of the bad data preprocessing method of the comprehensive energy system

[0166] The time spent by the AE-based bad data detection model and the GRU-Attention-based bad data identification and correction model in each run is shown in Tables 7 and 8. The single calculation time of the AE bad data detection model is 0.0001397s, which is reduced by 8.58% compared with random forest and 99.82% compared with SVM. The single calculation time of the GRU-Attention bad data detection model is 0.007893s, which is reduced by 88.61% and 27.96% compared with RNN and LSTM respectively, and increased by 2.71% compared with GRU. It can be seen that AE and GRU-Attention have certain calculation efficiency advantage compared with similar algorithms.

[0167] The calculation time of the AE-GRU-Attention bad data detection and identification correction model and the GRU-Attention bad data identification correction model in scenarios 1, 2, 3 and 4 is tested respectively, and the sum is taken as the test result, as shown in Table 9. It can be known from Tables 7 and 8 that the calculation time of the GRU-Attention bad data identification correction model is much greater than that of the AE bad data detection model. If only the GRU-Attention is relied on to directly identify the real-time measurement in actual operation, the calculation efficiency of the entire system will be low, and the existence of the AE bad data detection model can effectively improve the calculation efficiency of detection and identification correction and avoid waste of computing resources.

[0168] Table 7 Time used for single operation of the bad data detection model

[0169]

[0170] Table 8 Time used for single operation of the bad data identification correction model

[0171]

[0172]

[0173] Table 9 Time used for bad data detection and identification correction

[0174]

[0175] In order to analyze the calculation efficiency of the deep learning state estimation model combined with the application, the time used for state estimation of WLS, WLAV, CNN and the application in four situations of scenarios 1, 2, 3 and 4 is tested respectively, and the sum is taken as the test result, as shown in Table 10. After the application is combined with the CNN state estimation model, the time used for CNN state estimation is significantly increased. However, since the application belongs to a data-driven method, the calculation time complexity will not be doubled with the increase of the scale of the energy system, so although the application+CNN state estimation model has lower calculation efficiency than WLS in the state estimation of the power grid, gas network and heat network alone, the time used for joint estimation is much less than that of WLS. In addition, the calculation time of the application is much less than that of the robust state estimation method WLAV. The above situations fully reflect that the application has good calculation efficiency when combined with the deep learning state estimation model.

[0176] Table 10 Calculation time of the application for state estimation

[0177]

[0178] In the description of the specification, the description of the terms "one embodiment", "an example", "a specific example" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0179] The basic principles, main features and advantages of the present application are shown and described above. It should be understood by those skilled in the art that the present application is not limited by the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application.

Claims

1. A method for preprocessing adverse data in an integrated energy system, characterized in that, Includes the following steps: Collect real-time measurement data of the integrated energy system status and perform normalization processing; The normalized real-time measurement data is input into the integrated energy system bad data detection model. If the detection model determines that there is bad data in the measurement data, it will be identified and corrected; if the detection model determines that the measurement data is normal measurement data, the data preprocessing process will end. The location of the problematic data is determined and corrected using the integrated energy system's problematic data identification and correction model, and the data preprocessing process is then terminated. The integrated energy system defect data detection model is composed of multiple AE defect data detection models; AE includes encoders and decoders; The integrated energy system detrimental data identification and correction model is composed of multiple GRU-Attention detrimental data identification and correction models. The specific steps for identifying and correcting detrimental data using the integrated energy system detrimental data correction model include: S31 normalizes the historical measurement data of the integrated energy system, adds Gaussian white noise, and divides it into training set, validation set and test set; S32 divides the historical measurement data of the integrated energy system into multiple subsets according to categories, and builds various GRU-Attention bad data identification and correction models for each category of measurement data; and optimizes the model parameters by setting hyperparameters and training them multiple times. S33, input the validation set data into the trained GRU-Attention bad data identification and correction model, record the maximum relative error of the validation set prediction, and use it as the online identification threshold for bad data; S34: Read historical data of a certain type of measurement data containing bad data within a certain time period, perform normalization processing, and input it into the trained GRU-Attention bad data identification and correction model. Compare the predicted values ​​of the measurement data of each measurement point with the real-time measurement data. If the relative error is greater than the identification threshold, the measurement point is determined to be the location where bad data occurs. For the location where bad data occurs, replace it with the predicted value of the measurement data. The GRU-Attention bad data identification and correction model includes a GRU network and an Attention mechanism.

2. The method for preprocessing adverse data of an integrated energy system according to claim 1, characterized in that, The types of measurement data for integrated energy systems include: voltage amplitude at power grid nodes, active power of power grid branches, reactive power of power grid branches, active power at power grid nodes, reactive power at power grid nodes, thermal power at heating network nodes, pressure at heating network nodes, flow rate in heating network pipelines, supply water temperature at heating network nodes, return water temperature at heating network nodes, flow rate in gas network pipelines, pressure at gas network nodes, and injection flow rate at gas network nodes.

3. The method for preprocessing adverse data of an integrated energy system according to claim 1, characterized in that, The specific steps for using the integrated energy system defect data detection model to perform data assessment include: S21. Normalize the historical measurement data of the integrated energy system, add Gaussian white noise, and divide it into training set, validation set and test set. S22, the historical measurement data of the integrated energy system is divided into multiple subsets according to categories, and an AE bad data detection model is built for each category of measurement data. The reconstruction error is used as the loss function, and the parameters of the bad data detection models of each category are optimized through multiple training sessions. S23. Input the validation set data into the trained AE bad data detection model, calculate and determine the minimum reconstruction error of bad data, and use it as the bad data detection threshold. If the reconstruction error of a data is greater than the detection threshold, it is considered bad data; otherwise, it is normal data. Set the bad data thresholds from low to high, calculate the F1 score of the validation set at different thresholds, and select the case with the highest score as the online detection threshold for bad data. S24. Input the normalized real-time measurement data into the corresponding trained AE defective data detection model according to the category, calculate the reconstruction error of each type of measurement data under the current time segment, and compare it with the defective data detection threshold of the corresponding model. If it is greater than the detection threshold, it is determined that there is defective data in the real-time measurement data of that category; otherwise, it is determined to be a normal measurement.

4. The method for preprocessing adverse data of an integrated energy system according to claim 3, characterized in that, The encoder's calculation process is as follows: X=f encode (I)=σ encode (W encode I+b encode ) In the formula, I is the input vector of the autoencoder; f encode Here, X is the encoder function; W is the encoded information vector of the input; enocde b is the parameter vector of the encoder neuron; encode σ is the bias vector of the encoder neuron; encode is the activation function of the encoder neuron; The decoder's calculation process is as follows: O=f decode (X)=σ decode (W decode X+b decode ) In the formula, O is the output vector of the autoencoder; f decode For decoder functions; W deocde b is the parameter vector of the decoder neuron; decode σ is the bias vector of the decoder neuron; decode is the activation function of the decoder neuron; An autoencoder continuously updates neuron parameters during training to minimize the reconstruction error between input I and output O. The reconstruction error is calculated as follows: In the formula, e autoencoder Let be the reconstruction error of the autoencoder; J be the reconstruction loss function of the autoencoder, and be the mean squared error loss function; n be the dimension of the input vector; I i and O i These are the i-th components of the input vector and the output vector, respectively.

5. The method for preprocessing adverse data of an integrated energy system according to claim 3, characterized in that, The formula for calculating the F1 score is: In the formula, S precision S recall S F1 These represent the precision, recall, and F1 score of the sample, respectively; n TP n FP n TN n FN These represent the number of TP, FP, TN, and FN samples, respectively. TP represents correct positive examples, FP represents incorrect positive examples, TN represents correct negative examples, and FN represents incorrect negative examples, indicating that correct samples were predicted incorrectly. Here, bad data is considered as positive examples and normal data is considered as negative examples.

6. The method for preprocessing faulty data in an integrated energy system according to claim 1, characterized in that, The internal relationships within the GRU network are as follows: z t =σ(W z ·[h t-1 ,x t ]) r t =σ(W r ·[h t-1 ,x t ]) h' t =tanh(W h ·[r t *h t-1 ,x t ]) h t =(1-z t )*h t-1 +z t *h' t In the formula, z t For updating GRU cells; r t For updating GRU cells; x t h is the input signal of the GRU unit at the current moment. t-1 h t These are the outputs of the GRU unit at the previous and current time steps, respectively; W z W r W h σ represents the weight coefficients of each neuron in the GRU unit; σ is the activation function; and tanh is the hyperbolic tangent activation function.

7. The method for preprocessing adverse data of an integrated energy system according to claim 1, characterized in that, The basic mathematical expression of the attention mechanism is: s i =f score (u,v i ) c=a i v i f score (u,v i )=u T W s v i In the formula, u is the feature vector within the current time step; v i It is the feature vector with timestamp i in time series data; s i Attention score; f score The attention calculation function; α i The normalized attention score; c is based on attention scores and feature sequences v i The characteristics of the calculated current timestamp; W s These are the network parameters for the attention mechanism.

8. The method for preprocessing defective data in an integrated energy system according to claim 3, characterized in that, The training set contains only normal measurement data, while the validation set and test set consist of both normal measurement data and defective data. The defective data in the validation set and test set are constructed in the same way and in the same proportion.

9. A preprocessing system for defective data in an integrated energy system, comprising the method described in any one of claims 1-8, characterized in that, include: Data acquisition module: Collects real-time measurement data of the integrated energy system status and performs normalization processing; Defective data detection module: Input the normalized real-time measurement data into the integrated energy system defective data detection model. If the detection model determines that there is defective data in the measurement data, it will identify and correct it; if the detection model determines that the measurement data is normal measurement data, it will end the data preprocessing process. In addition, the bad data identification and correction module: uses the integrated energy system bad data identification and correction model to determine the location of bad data and correct it, and ends the data preprocessing process.

Images