A method and system for cleaning power data based on regular automatic encoder

Abstract

The present invention relates to a method and system for cleaning electric quantity data based on a regular autoencoder, the method comprising the following steps: S1, reading in the original data of the smart meter in the substation area; S2, calculating the daily electric quantity data of the user; S3, eliminating the abnormal daily electric quantity data of the user; S4, dividing the user electric quantity data into a cycle of 28 days, starting from Monday and ending on Sunday; S5, establishing a data filling model based on a regular autoencoder; S6, counting the number of missing days of the user's daily electric quantity, if the missing percentage is less than a set threshold, proceeding to step S7, otherwise, not filling the data for the user; S7, inputting the missing user data into the data filling model based on the regular autoencoder, correcting and filling the electric quantity data of the user; S8, filling the missing data. The method and system have a small error in the electric quantity data repair, and improve the data quality.

Description

Technical Field

[0001] The present invention belongs to the field of electric power big data application, and specifically relates to an electric power data cleaning method and system based on a regular autoencoder. Background Art

[0002] Power big data mainly comes from the power generation, transmission, transformation, distribution, consumption and dispatching of power production and electric energy use. By using smart terminal devices such as smart meters, various operation data of the power system can be collected. Then, the collected power big data can be systematically processed and analyzed to monitor, diagnose, optimize and predict the operation of the power grid, providing guarantee for the safe, reliable, economical and efficient operation of the power grid. Among them, the power data of electricity users is also an important component of power big data. However, due to the complexity of electricity consumption, there will be abnormalities and missing power data, which brings certain difficulties to data analysis. Therefore, it is necessary to process the abnormal and missing power data before data analysis. In the existing power data cleaning method, the data repair error is large, which affects the accuracy of big data analysis. Summary of the invention

[0003] The purpose of the present invention is to provide a method and system for cleaning electric quantity data based on a regular autoencoder, wherein the method and system have small errors in repairing electric quantity data and improve data quality.

[0004] To achieve the above object, the technical solution adopted by the present invention is: a method for cleaning power data based on a regular autoencoder, comprising the following steps:

[0005] S1, read the original data of the smart meter in the substation;

[0006] S2. Calculate the user's daily electricity consumption data;

[0007] S3. Eliminate abnormal daily electricity consumption data of users;

[0008] S4, dividing the user power consumption data into 28-day cycles, starting from Monday and ending on Sunday;

[0009] S5. Establish a data filling model based on regularized autoencoder;

[0010] S6. Count the number of days missing the user's daily electricity consumption. If the missing percentage is less than the set threshold, proceed to step S7. Otherwise, do not perform data filling for the user.

[0011] S7, inputting the missing user data into the data filling model based on the regular autoencoder to correct and fill in the power data of the user;

[0012] S8. Fill in the missing data.

[0013] Furthermore, the step S2 specifically includes the following steps:

[0014] S21, distinguishing electric meter data of different users according to user numbers;

[0015] S22, sorting the electric meter data of each user by date;

[0016] S23, determine whether the meter asset number of the next day is consistent with that of the previous day. If not, it means that the user has replaced the meter. The data displayed by the user's smart meter on that day is the user's daily electricity consumption. If consistent, execute step S24;

[0017] S24, determine whether the user's previous and next two days have values. If one or both are missing, set the user's power consumption for that day to blank, i.e., the data is missing. If both data exist, execute step S25;

[0018] S25. Subtract the positive active power of the smart meter from the previous day to obtain the user's daily power consumption data.

[0019] Furthermore, the step S3 specifically includes the following steps:

[0020] S31. According to the user's daily electricity consumption data obtained in step S2, if the data is a negative value, it is regarded as abnormal data, and the user's daily electricity consumption data is set to blank;

[0021] S32. According to the daily electricity consumption data of users obtained in step S2, if the electricity consumption data of all users on that day exist, but the sum value within the set error range is not equal to the total electricity consumption, it is determined that the electricity consumption data of all users on that day is wrong, and the electricity consumption data of all users on that day are set to blank.

[0022] Furthermore, the step S5 specifically includes the following steps:

[0023] S51. Regular constraints are imposed on the loss function of the autoencoder, and orthogonal constraints and L21 norms are embedded to realize the regularization of the autoencoder. The objective function expression of the regular autoencoder to be optimized is as follows:

[0024]

[0025] In formula (1), N is the total number of samples, x is the input data, is the reconstructed data, v is the number of layers of the regularized autoencoder, σ is the coefficient of the L21 norm, β is the coefficient of the orthogonal constraint, I is the identity matrix, and W is the weight of the model;

[0026] S52, establishing a training set of electricity consumption data including various types of users;

[0027] S53, using the training set to train the parameters of the regularized autoencoder; calculating the power consumption data x and the reconstructed data according to formula (1) The loss function is used to determine whether the error meets expectations. If so, the parameters of the regular autoencoder are saved. If not, the parameters are updated to perform encoding and decoding again.

[0028] S54: Save the best parameters of the regularized autoencoder to obtain a data filling model based on the regularized autoencoder.

[0029] Furthermore, in step S6, the threshold is set to 40% of the total number of days.

[0030] The present invention also provides an electric quantity data cleaning system based on a regular autoencoder, comprising a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0031] Compared with the prior art, the present invention has the following beneficial effects: it provides a method and system for cleaning power data based on a regularized autoencoder, which achieves regularization by adding the L21 norm and orthogonal constraints to the loss function through the regularized autoencoder, thereby preventing the model from overfitting and further improving the generalization ability of the model. The method can accurately extract the characteristics of the user's power consumption and fill in the missing data accordingly. The error of the data repair result is relatively small, which has a positive effect on the research on the correlation of power data and the improvement of data quality when big data technology is applied in the distribution network. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of the method implementation of the embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0034] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0035] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0036] like Figure 1 As shown, this embodiment provides a method for cleaning power data based on a regular autoencoder, comprising the following steps:

[0037] S1. Read the original data of the smart meter in the substation.

[0038] S2. Calculate the user's daily electricity consumption data.

[0039] The step S2 specifically includes the following steps:

[0040] S21, distinguishing electric meter data of different users according to user numbers;

[0041] S22, sorting the electric meter data of each user by date;

[0042] S23, determine whether the meter asset number of the next day is consistent with that of the previous day. If not, it means that the user has replaced the meter. The data displayed by the user's smart meter on that day is the user's daily electricity consumption. If consistent, execute step S24;

[0043] S24, determine whether the user's previous and next two days have values. If one or both are missing, set the user's power consumption for that day to blank, i.e., the data is missing. If both data exist, execute step S25;

[0044] S25. Subtract the positive active power of the smart meter from the previous day to obtain the user's daily power consumption data.

[0045] S3. Eliminate abnormal daily electricity consumption data of users.

[0046] The step S3 specifically comprises the following steps:

[0047] S31. According to the user's daily electricity consumption data obtained in step S2, if the data is a negative value, it is regarded as abnormal data, and the user's daily electricity consumption data is set to blank;

[0048] S32. According to the daily electricity consumption data of users obtained in step S2, if the electricity consumption data of all users on that day exist, but the sum value within the set error range is not equal to the total electricity consumption, it is determined that the electricity consumption data of all users on that day is wrong, and the electricity consumption data of all users on that day are set to blank.

[0049] S4. The user power consumption data is divided into 28-day cycles, starting from Monday and ending on Sunday.

[0050] S5. Establish a data filling model based on regularized autoencoder.

[0051] The step S5 specifically comprises the following steps:

[0052] S51. Regular constraints are imposed on the loss function of the autoencoder, and orthogonal constraints and L21 norms are embedded to realize the regularization of the ordinary autoencoder. The objective function expression of the regular autoencoder to be optimized is as follows:

[0053]

[0054] In formula (1), N is the total number of samples, x is the input data, is the reconstructed data, v is the number of layers of the regularized autoencoder, σ is the coefficient of the L21 norm, β is the coefficient of the orthogonal constraint, I is the identity matrix, and W is the weight of the model;

[0055] S52, establishing a training set of electricity consumption data including various types of users;

[0056] S6. Count the number of days with missing data on the user's daily electricity consumption. If the missing percentage is less than a set threshold, proceed to step S7. Otherwise, do not perform data filling for the user.

[0057] In this embodiment, the threshold is set to 40% of the total number of days.

[0058] S7. Input the missing user data into the data filling model based on the regular autoencoder to correct and fill in the power data of the user.

[0059] S8. Fill in the missing data.

[0060] S53, using the training set to train the parameters of the regularized autoencoder; calculating the power consumption data x and the reconstructed data according to formula (1) The loss function is used to determine whether the error meets expectations. If so, the parameters of the regular autoencoder are saved. If not, the parameters are updated to perform encoding and decoding again.

[0061] S54: Save the best parameters of the regularized autoencoder to obtain a data filling model based on the regularized autoencoder.

[0062] This embodiment also provides an electric quantity data cleaning system based on a regular autoencoder, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0063] In this embodiment, this method and the method based on the ordinary autoencoder and the denoising autoencoder are used to perform data filling respectively, and the effects are shown in Table 1.

[0064] Table 1 Comparison of data filling effects between this method and existing technology

[0065]

[0066] In Table 1, the E function is expressed as:

[0067]

[0068] In formula (2), x ij is the true value, x i ' j To fill in the values, M is the set of all missing values.

[0069] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0070] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0071] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0072] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0073] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A method for cleaning power data based on a regular autoencoder, characterized in that: The following steps are involved: S1, read the original data of the smart meter in the substation; S2. Calculate the user's daily electricity consumption data; S3. Eliminate abnormal daily electricity consumption data of users; S4, dividing the user power consumption data into 28-day cycles, starting from Monday and ending on Sunday; S5. Establish a data filling model based on regularized autoencoder; S6. Count the number of days missing the user's daily electricity consumption. If the missing percentage is less than the set threshold, proceed to step S7. Otherwise, do not perform data filling for the user. S7, inputting the missing user data into the data filling model based on the regular autoencoder to correct and fill in the power data of the user; S8, fill in missing data; The step S5 specifically comprises the following steps: S51. Regular constraints are imposed on the loss function of the autoencoder, and orthogonal constraints and L21 norms are embedded to realize the regularization of the autoencoder. The objective function expression of the regular autoencoder to be optimized is as follows: In formula (1), N is the total number of samples, x is the input data, is the reconstructed data, v is the number of layers of the regularized autoencoder, σ is the coefficient of the L21 norm, β is the coefficient of the orthogonal constraint, I is the identity matrix, and W is the weight of the model; S52, establishing a training set of electricity consumption data including various types of users; S53, using the training set to train the parameters of the regularized autoencoder; calculating the power consumption data x and the reconstructed data according to formula (1) The loss function is used to determine whether the error meets expectations. If so, the parameters of the regular autoencoder are saved. If not, the parameters are updated to perform encoding and decoding again. S54: Save the best parameters of the regularized autoencoder to obtain a data filling model based on the regularized autoencoder.

2. The method for cleaning electric quantity data based on a regular autoencoder according to claim 1, characterized in that: The step S2 specifically includes the following steps: S21, distinguishing electric meter data of different users according to user numbers; S22, sorting the electric meter data of each user by date; S23, determine whether the meter asset number of the next day is consistent with that of the previous day. If not, it means that the user has replaced the meter. The data displayed by the user's smart meter on that day is the user's daily electricity consumption. If consistent, execute step S24; S24, determine whether the user's previous and next two days have values. If one or both are missing, set the user's power consumption for that day to blank, i.e., the data is missing. If both data exist, execute step S25; S25. Subtract the positive active power of the smart meter from the previous day to obtain the user's daily power consumption data.

3. The method for cleaning electric quantity data based on regular autoencoder according to claim 1, characterized in that: The step S3 specifically comprises the following steps: S31. According to the user's daily electricity consumption data obtained in step S2, if the data is a negative value, it is regarded as abnormal data, and the user's daily electricity consumption data is set to blank; S32. According to the daily electricity consumption data of users obtained in step S2, if the electricity consumption data of all users on that day exist, but the sum value within the set error range is not equal to the total electricity consumption, it is determined that the electricity consumption data of all users on that day is wrong, and the electricity consumption data of all users on that day are set to blank.

4. The method for cleaning electric quantity data based on regular autoencoder according to claim 1, characterized in that: In step S6, the threshold is set to 40% of the total number of days.

5. A power data cleaning system based on regular autoencoder, characterized in that: The method comprises a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps as described in any one of claims 1 to 4 can be implemented.

Images