Power transaction settlement data cleaning and verification method and system, and computer equipment
Through deep learning technology, the construction of a power transaction settlement data cleaning and verification model has solved the problem that traditional methods cannot effectively identify abnormal power transaction data, and achieved more efficient and accurate data cleaning, improving data quality and reliability of settlement results.
Patent Information
- Application Number
- CN202411627164.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-14
- Publication Date
- 2025-03-07
AI Technical Summary
Traditional abnormal data detection methods cannot fully consider the timing characteristics and periodic changes of power transaction data, and it is difficult to effectively identify complex data abnormalities, and perform poorly when dealing with data loss and noise.
Deep learning technology is used to build a power transaction settlement data cleaning and verification model, and abnormal samples are found and detected through unsupervised learning, and data cleaning and verification are carried out in combination with expert knowledge. The model includes a noisy input layer, a feature extraction network layer, a sample reconstruction network layer and an output layer, and is constructed and trained using the TensorFlow platform.
It significantly improves the efficiency and accuracy of data cleaning, can more effectively handle the timing characteristics and noise characteristics in the power transaction settlement data, and improves the data quality and reliability of settlement results.
Smart Images
Figure CN119577329A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of power settlement data preprocessing, and in particular to a method, system and computer equipment for cleaning and verifying power transaction settlement data. Background Art
[0002] As market-oriented reforms in power trading continue to deepen, the power trading settlement process has become a key component in ensuring fairness and transparency in power market transactions. Power trading settlement is the final step in the power market and a crucial economic accounting process in market operations. The accuracy of power trading settlement directly impacts the economic interests of market participants and the health of the market. Therefore, fully ensuring and verifying the accuracy and reliability of power trading settlement data is crucial.
[0003] Power trading settlement data typically includes information on power generation, traded electricity, traded electricity prices, and electricity consumption. This data is not only massive but also exhibits complex time series characteristics and dynamic changes. To ensure the accuracy of settlement results, data must be analyzed, cleansed, and verified. In the actual power trading settlement process, data noise and anomalies are inevitable, and these noise and anomalies can seriously impact the accuracy of settlement results.
[0004] While common anomaly data detection methods, such as the 3σ criterion, principal component analysis (PCA), and the k-nearest neighbor (kNN) algorithm, have been widely used in many data cleaning tasks, they exhibit limitations when processing power trading settlement data. These anomaly data detection methods fail to fully account for the temporal characteristics and cyclical variations of power trading data, making it difficult to effectively identify complex data anomalies. Furthermore, these methods are limited in their ability to handle missing data and data noise, resulting in unsatisfactory performance in anomaly detection and data cleaning for power trading settlement data. Summary of the Invention
[0005] In response to the shortcomings of the existing technology, the present invention provides a method, system and computer equipment for cleaning and verifying electricity trading settlement data, which solves the technical problem that traditional abnormal data detection methods cannot fully consider the timing characteristics and periodic changes of electricity trading data, and have difficulty in effectively identifying complex data anomalies.
[0006] To solve the above technical problems, the present invention provides the following technical solution: a method for cleaning and verifying power transaction settlement data, the method comprising the following steps:
[0007] S1. Obtain electricity settlement data samples and construct a training set X based on the electricity settlement data samples train , validation set X validand the test set X test ;
[0008] S2, use the range method to analyze the training set X train , validation set X valid and the test set X test Perform normalization to obtain the normalized training set X' train , validation set X' valid and test set X' test ;
[0009] S3: Build a power settlement data cleaning and verification model based on the open source deep learning platform TensorFlow;
[0010] S4, using training set X' train Train the electricity settlement data cleaning and verification model, and define the loss function J to iteratively update the electricity settlement data cleaning and verification model;
[0011] S5. Use the validation set X' valid and test set X' test Conduct verification, evaluation and testing on the updated electricity settlement data cleaning and verification models;
[0012] S6. Use the electricity settlement data as input to the electricity settlement data cleaning and verification model and output abnormal data, and remove the abnormal data from the electricity settlement data.
[0013] Furthermore, in step S1, specifically: collect n tr Electricity settlement data samples are used as training set X train ∈R t×d ; n without data cleaning and verification va Electricity settlement data samples are used as the validation set X valid ∈R t×d , which contains e va abnormal data samples; n data samples that have not been cleaned and verified te Electricity settlement data samples are used as the test set X test ∈R t×d ; where t represents the length of the sample time series, and d represents the dimension of the variable at a certain moment.
[0014] Furthermore, in step S3, the electricity settlement data cleaning and verification model includes:
[0015] It is used to add random noise to the input power settlement data sample X and obtain the sample X noised The noisy input layer, and
[0016] The sample X noisedMapped to the hidden space to obtain the feature extraction network layer of the latent variable z, and
[0017] The latent variable z is decoded by the decoder to obtain the reconstructed sample The sample reconstruction network layer,
[0018] And, through the Reshape operation, several reconstructed samples An output layer that performs the reorganization and outputs the reconstructed data.
[0019] Furthermore, the feature extraction network layer and the sample reconstruction network layer are both composed of a number of fully connected layers and ReLu activation functions connected in series, and the first and last layers of the feature extraction network layer and the sample reconstruction network layer are both fully connected layers.
[0020] Furthermore, in step S4, the specific process includes the following steps:
[0021] S41, Flatten the training set X' train The electricity settlement data samples in the grid are tiled, and random noise is added to the electricity settlement data sample input through the Dropout operation to obtain the sample X noised ;
[0022] S42, the noised sample X noised The encoder is mapped to the hidden space to obtain the hidden variable z, and the network parameter of the encoder is defined as θ e ;
[0023] S43, decode the latent variable z through the decoder to obtain the reconstructed sample And define the network parameters of the decoder as θ d ;
[0024] S44. Define the loss function of the power settlement data cleaning and verification model as J, where
[0025]
[0026] Where X is the electricity settlement data sample; To reconstruct the sample;
[0027] S45, through the stochastic gradient descent iterative algorithm, the network parameters of the feature extraction network layer and the sample reconstruction network layer are n epoch Iterative update.
[0028] Furthermore, in step S45, the updating method is as follows:
[0029] The network parameters of the encoder are updated as follows:
[0030] The network parameters of the decoder are updated as follows:
[0031] Where μ0 is the learning rate and J is the loss function of the electricity settlement data cleaning and verification model.
[0032] Furthermore, in step S5, the specific process includes the following steps:
[0033] S51, calculate each input sample x i ∈X' train The reconstruction root mean square error RMSE i , the calculation formula is:
[0034]
[0035] Among them, t represents the length of the sample time series; d represents the dimension of the variable at a certain moment; For the training set X' train The jth input sample x in i ; For the jth input sample x i Corresponding reconstructed samples;
[0036] S52, according to the reconstruction root mean square error RMSE i Calculate for all input samples x i The mean μ and standard deviation σ are calculated as follows:
[0037]
[0038] Among them, n tr For the training set X' train The number of electricity settlement data samples that have undergone data cleaning and verification;
[0039] S53, the verification set X' valid Input into the electricity settlement data cleaning and verification model to calculate each verification set sample x i ∈X' valid The reconstruction root mean square error RMSE i ;
[0040] If μ-kσ≤RMSE i ≤μ+kσ, then the validation set sample x i Recorded as a normal sample, let the power settlement data f i =0;
[0041] On the contrary, the validation set sample x i Recorded as abnormal samples, let the power settlement data f i =1, where k is a constant;
[0042] S54. Calculate the anomaly detection rate p used to evaluate the performance of the power settlement data cleaning and verification model valid ;
[0043] S55, using the test set X' test Test the electricity settlement data cleaning and verification model.
[0044] Furthermore, in step S54, the abnormality detection rate p valid The calculation formula is:
[0045]
[0046] Among them, e va For the validation set X' valid The number of abnormal data samples in n va Validation set X' valid The number of electricity settlement data samples that have not been cleaned and verified.
[0047] Through the above technical solution, the present invention provides a method, system and computer equipment for cleaning and verifying power transaction settlement data, which have at least the following beneficial effects:
[0048] 1. This paper combines the temporal characteristics and noise characteristics of electricity settlement data to construct an electricity settlement data cleaning and verification model. Through unsupervised learning methods, it discovers and detects abnormal samples in electricity settlement data, and combines expert knowledge to achieve electricity settlement data cleaning and verification.
[0049] 2. The method proposed in the present invention can not only fully consider the periodic characteristics of power transaction settlement data, but also has the ability to deal with data missing and noise problems. Through deep learning technology, it can automatically extract data features and detect anomalies, which can significantly improve the efficiency and accuracy of data cleaning, thereby improving the data quality and reliability of settlement results in the power transaction settlement process. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0051] Figure 1 This is a flow chart of the method for cleaning and verifying power transaction settlement data of the present invention;
[0052] Figure 2 This is a network structure diagram of the power settlement data cleaning and verification model of the present invention. DETAILED DESCRIPTION
[0053] To make the above-mentioned objectives, features, and advantages of the present invention more clearly understood, the present invention is further described below in detail with reference to the accompanying drawings and specific embodiments. This will enable a full understanding of how this application uses technical means to solve technical problems and achieve technical effects, and to implement the invention accordingly.
[0054] Please refer to Figure 1-Figure 2 , showing a specific implementation of this embodiment. This embodiment addresses the problem of power transaction settlement data cleansing and verification, combining the time series and noise characteristics of power settlement data to propose a power transaction settlement data cleansing and verification method, system, and computer equipment. The following further illustrates the specific process and effectiveness of the power transaction settlement data cleansing and verification method, using actual electricity consumption data used in the power settlement process. Actual electricity consumption data is collected over a 7-day period, from 00:00 each day to 00:00 the following day. The cumulative electricity consumption for the previous 6 hours is calculated every 6 hours, resulting in electricity consumption data being collected four times per day.
[0055] Please refer to Figure 1 This embodiment proposes a method for cleaning and verifying power transaction settlement data. This method fully considers the periodic characteristics of power transaction settlement data, is capable of handling data missing and noise issues, and uses deep learning technology to automatically extract data features and detect anomalies. The method includes the following steps:
[0056] S1. Obtain electricity settlement data samples and construct a training set X based on the electricity settlement data samples train , validation set X valid and the test set X test ; Specifically: collect n tr Electricity settlement data samples are used as training set X train ∈R t×d ; n without data cleaning and verification va Electricity settlement data samples are used as the validation set X valid ∈R t ×d , which contains e va abnormal data samples; n data samples that have not been cleaned and verified te Electricity settlement data samples are used as the test set X test ∈R t×d ; where t represents the length of the sample time series, and d represents the dimension of the variable at a certain moment.
[0057] In this embodiment, according to the characteristics of power consumption data, the time series data length t=7 and the sample variable dimension d=4 are taken. tr = 10,000 cleaned and verified electricity consumption time series data samples as training set, n va= 1000 samples of uncleaned and unverified electricity consumption time series data as the validation set, which contains e va = 20 abnormal samples, n te = 200 samples of uncleaned and unverified electricity consumption time series data are used as the test set.
[0058] S2, use the range method to analyze the training set X train , validation set X valid and the test set X test Perform normalization to obtain the normalized training set X' train , validation set X' valid and test set X' test , which is used as the input for the subsequent construction of the power settlement data cleaning and verification model. Specifically, the training set X train As an example, first select the training set X train The minimum value X of the power settlement data sample min , maximum value X max , and then define the normalization function as:
[0059]
[0060] Where X is the electricity settlement data sample.
[0061] Finally, normalize the training set, validation set, and test set in turn to obtain X' train =f(X train ), X' valid =f(X valid ), X' test =f(X test ).
[0062] S3. Building a power settlement data cleaning and verification model based on the open source deep learning platform TensorFlow; In step S3, the power settlement data cleaning and verification model includes: adding random noise to the input power settlement data sample X to obtain the sample X noised The noisy input layer, and the sample X noised Mapping to the hidden space to obtain the feature extraction network layer of the latent variable z, and decoding the latent variable z through the decoder to obtain the reconstructed sample The sample reconstruction network layer, and the Reshape operation to reconstruct several samples The output layer performs reorganization and outputs the reconstructed data. Among them, the feature extraction network layer and the sample reconstruction network layer are both composed of several fully connected layers and ReLu activation functions connected in series, and the first and last layers of the feature extraction network layer and the sample reconstruction network layer are both fully connected layers.
[0063] In this embodiment, the network structure of the power settlement data cleaning and verification model is as follows: Figure 2 As shown. The input data is flattened, i.e., the Flatten operation, and random noise is added through the Dropout operation, finally obtaining a noisy input layer with a size of 7×4=28. The encoder network is defined as two fully connected layers and a ReLu activation layer connected in series, with the order as follows Figure 2 The encoder part is shown in the figure, where the sizes of the two fully connected layers are defined as 40 and 20 respectively. The noisy input layer is mapped by the encoder network to obtain the latent variable in the hidden layer space, whose size is defined as 20. The decoder network is defined as two fully connected layers and a ReLu activation layer connected in series, and the order is as follows: Figure 2 As shown in the decoder, the sizes of the two fully connected layers are defined as 20 and 40, respectively. The output layer is then passed through a fully connected layer with a size of 28. The output layer data is reshaped into a 7×4 size, i.e., the reshape operation, to obtain the final reconstructed data.
[0064] S4, using training set X' train The power settlement data cleaning and verification model is trained, and the loss function J is defined to iteratively update the power settlement data cleaning and verification model; the normalized training set X' is used train The power settlement data cleaning and verification model is trained with a learning rate of μ0 = 0.01 and an iteration number of n. epoch =500, each iteration evenly divides the input sample into small sample groups of length 20 and inputs them into the model for training, calculates the model's loss function J and updates the network parameters θ e ,θ d In step S4, the specific process includes the following steps:
[0065] S41, Flatten the training set X' train The electricity settlement data samples in the grid are tiled, and random noise is added to the electricity settlement data sample input through the Dropout operation to obtain the sample X noised ;
[0066] S42, the noised sample X noised The encoder is mapped to the hidden space to obtain the hidden variable z, and the network parameter of the encoder is defined as θ e ;
[0067] S43, decode the latent variable z through the decoder to obtain the reconstructed sample And define the network parameters of the decoder as θ d ;
[0068] S44. Define the loss function of the power settlement data cleaning and verification model as J, where
[0069]
[0070] Where X is the electricity settlement data sample; To reconstruct the sample;
[0071] S45, through the stochastic gradient descent iterative algorithm, the network parameters of the feature extraction network layer and the sample reconstruction network layer are n epoch The update is done iteratively as follows:
[0072] The network parameters of the encoder are updated as follows:
[0073] The network parameters of the decoder are updated as follows:
[0074] Where μ0 is the learning rate and J is the loss function of the electricity settlement data cleaning and verification model.
[0075] S5. Use the validation set X' valid and test set X' test The updated electricity settlement data cleaning and verification model is verified, evaluated, and tested respectively; in step S5, the specific process includes the following steps:
[0076] S51, calculate each input sample x i ∈X' train The reconstruction root mean square error RESE i , the calculation formula is:
[0077]
[0078] Among them, t represents the length of the sample time series; d represents the dimension of the variable at a certain moment; For the training set X' train The jth input sample x in i ; For the jth input sample x i Corresponding reconstructed samples;
[0079] S52, according to the reconstruction root mean square error RMSE i Calculate for all input samples x i The mean μ and standard deviation σ are calculated as follows:
[0080]
[0081] Among them, n tr For the training set X' train The number of electricity settlement data samples that have undergone data cleaning and verification;
[0082] S53, the verification set X' valid Input into the electricity settlement data cleaning and verification model to calculate each verification set sample x i ∈X' valid The reconstruction root mean square error RMSE i ;
[0083] If μ-kσ≤RMSE i ≤μ+kσ, then the validation set sample x i Recorded as a normal sample, let the power settlement data f i =0;
[0084] On the contrary, the validation set sample x i Recorded as abnormal samples, let the power settlement data f i =1, where k is a constant;
[0085] S54. Calculate the anomaly detection rate p used to evaluate the performance of the power settlement data cleaning and verification model valid , where p valid The closer it is to 1, the better the model effect is. The calculation formula is:
[0086]
[0087] Among them, e va For the validation set X' valid The number of abnormal data samples in n va Validation set X' valid The number of electricity settlement data samples that have not been cleaned and verified is in the middle;
[0088] S55, using the test set X' test Test the electricity settlement data cleaning and verification model.
[0089] In this embodiment, the normalized validation set X' valid Input into the trained electricity settlement data cleaning and verification model, and calculate each verification set sample x i ∈X' valid The reconstruction root mean square error RMSE i If μ-kσ≤RMSE i ≤μ+kσ, then x i Recorded as a normal sample, let f i =0, otherwise x i Denote it as an abnormal sample, let f i =1, take k=3. Calculate the abnormality detection rate of the validation set Table 1 compares the detection rates of the proposed model and other common anomaly detection algorithms on the validation set.
[0090] Table 1 Comparison of detection rates of different models
[0091] Indicators / Models 3σ criterion PCA kNN (k=3) The model of the present invention Detection rate 0.85 0.90 0.93 0.98
[0092] From the detection rate comparison results shown in Table 1, it can be seen that the power settlement data cleaning and verification model proposed in the present invention has the highest detection rate for abnormal data in the power settlement data, and therefore has a better cleaning effect on the power settlement data.
[0093] S6. Use the electricity settlement data as the input of the electricity settlement data cleaning and verification model and output abnormal data, and remove the abnormal data from the electricity settlement data. Specifically, the electricity settlement data cleaning and verification model can find all abnormal data and submit them for manual confirmation. Data confirmed as abnormal will be removed from the electricity settlement data.
[0094] This embodiment proposes a power settlement data cleaning and verification system, which includes:
[0095] The data set construction module is used to obtain power settlement data samples and construct a training set X based on the power settlement data samples. train , validation set X valid and the test set X test ;
[0096] Normalization module uses the range method to normalize the training set X train , validation set X valid and the test set X test Perform normalization to obtain the normalized training set X' train , validation set X' valid and test set X' test ;
[0097] A model building module that builds a power settlement data cleaning and verification model based on the open source deep learning platform TensorFlow;
[0098] Training update module, using training set X' train Train the electricity settlement data cleaning and verification model, and define the loss function J to iteratively update the electricity settlement data cleaning and verification model;
[0099] Verification test module, using verification set X' valid and test set X' test Conduct verification, evaluation and testing on the updated electricity settlement data cleaning and verification models;
[0100] The output module is used to use the electricity settlement data as the input of the electricity settlement data cleaning and verification model and output abnormal data, thereby removing the abnormal data from the electricity settlement data.
[0101] The present invention combines the temporal characteristics and noise characteristics of electricity settlement data to construct an electricity settlement data cleaning and verification model. Through unsupervised learning methods, it discovers and detects abnormal samples in electricity settlement data, and combines expert knowledge to achieve electricity settlement data cleaning and verification.
[0102] This embodiment also provides a computer device comprising a processor, memory, a network interface, an input device, and a display screen connected via a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory includes a storage medium and internal memory. The storage medium can be either non-volatile or volatile. The storage medium stores an operating system and may also store computer-readable instructions. When executed by the processor, the computer-readable instructions enable the processor to implement a method for cleansing and verifying power trading settlement data. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the storage medium. The internal memory may also store computer-readable instructions. When executed by the processor, the computer-readable instructions enable the processor to implement the method for cleansing and verifying power trading settlement data. The network interface of the computer device is configured to communicate with an external server via a network connection. The display screen of the computer device can be a liquid crystal display or an electronic ink display. The input device of the computer device can be a touchscreen covering the display screen, buttons, a trackball, or a touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse.
[0103] Those skilled in the art will appreciate that all or part of the steps in the above-mentioned embodiment methods can be accomplished by instructing the relevant hardware through a program. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0104] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to in detail. For the above embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For relevant parts, please refer to the partial description of the method embodiments.
[0105] The above embodiments provide a detailed introduction to the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.
Claims
1. A method for cleaning and verifying power transaction settlement data, characterized in that: The method comprises the following steps: S1. Obtain power settlement data samples and construct a training set X based on the power settlement data samples train , validation set X valid and the test set X test ; S2, use the range method to train the training set X train , validation set X valid and the test set X test Perform normalization to obtain the normalized training set X' train , validation set X' valid and test set X' test ; S3, build a power settlement data cleaning and verification model based on the open source deep learning platform TensorFlow; S4, using training set X' train Train the power settlement data cleaning and verification model, and define the loss function J to iteratively update the power settlement data cleaning and verification model; S5. Use the validation set X' valid and test set X' test Verify, evaluate and test the updated power settlement data cleaning and verification models respectively; S6. Use the electricity settlement data as the input of the electricity settlement data cleaning and verification model and output the abnormal data, and remove the abnormal data from the electricity settlement data.
2. The method for cleaning and verifying power transaction settlement data according to claim 1, characterized in that: In step S1, specifically: collect n data that have been cleaned and verified tr Electricity settlement data samples are used as training set X train ∈R t ×d ; Data that has not been cleaned and verified va Electricity settlement data samples are used as the validation set X valid ∈R t×d , which contains e va abnormal data samples; n data samples that have not been cleaned and verified te The power settlement data samples are used as the test set X test ∈R t×d ; where t represents the length of the sample time series, and d represents the dimension of the variable at a certain moment.
3. The method for cleaning and verifying power transaction settlement data according to claim 1, characterized in that: In step S3, the power settlement data cleaning and verification model includes: It is used to add random noise to the input power settlement data sample X to obtain the sample X noised The noisy input layer, and The sample X noised Mapped to the hidden space to obtain the feature extraction network layer of the hidden variable z, and The latent variable z is decoded by the decoder to obtain the reconstructed sample The sample reconstruction network layer, And, through the Reshape operation, several reconstructed samples An output layer that performs the reorganization and outputs the reconstructed data.
4. The method for cleaning and verifying power transaction settlement data according to claim 3, characterized in that: The feature extraction network layer and the sample reconstruction network layer are both composed of a number of fully connected layers and ReLu activation functions connected in series at intervals, and the first and last layers of the feature extraction network layer and the sample reconstruction network layer are both fully connected layers.
5. The method for cleaning and verifying power transaction settlement data according to claim 1, characterized in that: In step S4, the specific process includes the following steps: S41, Flatten the training set X' train The power settlement data samples in the grid are tiled, and random noise is added to the power settlement data sample input through the Dropout operation to obtain the sample X noised ; S42, the noise-added sample X noised The encoder is mapped to the hidden space to obtain the hidden variable z, and the network parameter of the encoder is defined as θ e ; S43, decode the latent variable z through the decoder to obtain the reconstructed sample And define the network parameters of the decoder as θ d ; S44. Define the loss function of the power settlement data cleaning and verification model as J, where: Among them, X is the electricity settlement data sample; To reconstruct the sample; S45, through the stochastic gradient descent iterative algorithm, the network parameters of the feature extraction network layer and the sample reconstruction network layer are n epoch Iteration update.
6. The method for cleaning and verifying power transaction settlement data according to claim 5, characterized in that: In step S45, the updating method is as follows: The network parameters of the encoder are updated as: The network parameters of the decoder are updated as: Among them, μ0 is the learning rate; J is the loss function of the power settlement data cleaning and verification model.
7. The method for cleaning and verifying power transaction settlement data according to claim 1, characterized in that: In step S5, the specific process includes the following steps: S51, calculate each input sample x i ∈X' train The reconstruction root mean square error RMSE i , the calculation formula is: Among them, t represents the length of the sample time series; d represents the dimension of the variable at a certain moment; For the training set X' train The jth input sample x in i ; For the jth input sample x i The corresponding reconstructed samples; S52, according to the reconstruction root mean square error RMSE i Calculate for all input samples x i The mean value μ and standard deviation σ are calculated as follows: Among them, n tr For the training set X' train The number of electricity settlement data samples that have been cleaned and verified; S53, the verification set X' valid Input into the power settlement data cleaning and verification model to calculate each verification set sample x i ∈X' valid The reconstruction root mean square error RMSE i ; If μ-kσ≤RMSE i ≤μ+kσ, then the validation set sample x i Recorded as a normal sample, let the power settlement data f i =0; On the contrary, the validation set sample x i Recorded as abnormal samples, let the power settlement data f i =1, where k is a constant; S54. Calculate the anomaly detection rate p used to evaluate the performance of the power settlement data cleaning and verification model valid ; S55, using the test set X' test Test the electricity settlement data cleaning and verification model.
8. The method for cleaning and verifying power transaction settlement data according to claim 7, characterized in that: In step S54, the abnormality detection rate p valid The calculation formula is: Among them, e va For the validation set X' valid The number of abnormal data samples in; n va Validation set X' valid The number of electricity settlement data samples that have not been cleaned and verified.
9. A system for implementing the method for cleaning and verifying power transaction settlement data according to any one of claims 1 to 8, characterized in that: The system includes: The data set construction module is used to obtain power settlement data samples and construct a training set X based on the power settlement data samples. train , validation set X valid and the test set X test ; The normalization module uses the range method to normalize the training set X train , validation set X valid and the test set X test Perform normalization to obtain the normalized training set X' train , validation set X' valid and test set X' test ; Model building module, which builds a power settlement data cleaning and verification model based on the open source deep learning platform TensorFlow; Training update module, using training set X' train Train the power settlement data cleaning and verification model, and define the loss function J to iteratively update the power settlement data cleaning and verification model; Verification test module, using verification set X' valid and test set X' test Verify, evaluate and test the updated power settlement data cleaning and verification models respectively; The output module is used to use the electricity settlement data as the input of the electricity settlement data cleaning and verification model and output abnormal data, so as to remove the abnormal data from the electricity settlement data.
10. A computer device, characterized in that: It comprises a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method for cleaning and verifying power transaction settlement data as described in any one of claims 1 to 8 is implemented.