A power time series data real-time cleaning method and system

Abstract

The application provides a power time sequence data real-time cleaning method and system. The method comprises the following steps: constructing a data anomaly judgment temporary table by using an attribute Id, a Key and historical data; obtaining real-time time sequence data which has passed protocol analysis and is transmitted by an IOT device in real time; after the time sequence data is received by a message middleware Kafka, a Flink component obtains the real-time time sequence data in real time, compares and calculates the real-time time sequence data with the data anomaly judgment temporary table, obtains the validity of the real-time time sequence data for judgment, and applies the judgment result to field identification of the data. The scheme provided by the application realizes real-time data anomaly cleaning by constructing a data anomaly judgment temporary table and dynamically updating, saves real-time computing resources, and meets the performance requirement under a high-speed data flow.

Description

Technical Field

[0001] The present invention belongs to the field of power data preprocessing, and in particular relates to a real-time cleaning method and system for power time series data. Background Art

[0002] With the development of the Industrial Internet, the application of information and digital technologies in the production processes of power companies is becoming increasingly widespread. This requires power companies to collect large amounts of time-series data from both the user side and the grid side. Simultaneously, with the advancement of infrastructure such as data collection devices and communication networks, the frequency, granularity, and accuracy of data collection have improved. However, due to factors such as electromagnetic interference, a large number of data quality issues, such as missing values ​​and outliers, still exist.

[0003] In particular, systems that display and apply power data require real-time data, that is, the results to be displayed or applied must be updated in real time with the frequency of data transmission. Therefore, the collected power data needs to be cleaned in real time. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention proposes a technical solution for a real-time cleaning method of power time series data to solve the above technical problems.

[0005] A first aspect of the present invention discloses a method for real-time cleaning of power time series data, the method comprising:

[0006] Step S1: Obtain the attribute ID of the pre-received time series data and the indicator name Key of the time series data, classify the preset incremental data or preset random data for the indicator name Key of each time series data, use the attribute ID and the indicator name Key of the data as query conditions, retrieve the power history data, and construct a data anomaly temporary table including: attribute ID, indicator name Key of the data, classification, mean, standard deviation, time and Value;

[0007] Step S2: Receive real-time time series data with a data structure of attribute ID, time and data indicator name Key and Value;

[0008] Step S3: query the data anomaly temporary table based on the attribute ID and the indicator name Key of the data in the received real-time time series data, determine the classification of the data in the queried data anomaly temporary table, and calculate the data anomaly identification statistic Z based on the classification by applying the Value of the real-time time series data, the Value of the data anomaly temporary table, the time of the real-time time series data, the time of the data anomaly temporary table, the mean and the standard deviation;

[0009] Step S4: If the data anomaly identification statistic Z is greater than a preset threshold, the real-time time series data is abnormal data; otherwise, the real-time time series data is normal data.

[0010] According to the method of the first aspect of the present invention, in step S1, the method of classifying the indicator name Key of each time series data into preset incremental data or preset random data includes:

[0011] If the data value continues to increase with time until it reaches the maximum value of the data and then increases again from 0, the data is defined as preset incremental data;

[0012] If the data values ​​do not have monotonicity as time increases and appear to be disordered, the data is defined as preset random data.

[0013] According to the method of the first aspect of the present invention, in step S1, the method of constructing a temporary data anomaly table including attribute Id, data indicator name Key, classification, mean, standard deviation, time, and Value includes:

[0014] Step S11: Obtain the attribute ID of the pre-received time series data and the index name of the data, i.e., the index name Key;

[0015] Step S12: classify the indicator name Key of each data into preset incremental data or preset random data;

[0016] Step S13: Using the attribute ID and the data indicator name Key as query conditions, retrieve one year's electricity history data;

[0017] Step S14: Calculate the mean and standard deviation based on the preset incremental data or the data type of the preset random data of the indicator name Key of each data of the pre-received time series data, and fill them into the mean and standard deviation fields in the data anomaly temporary table accordingly;

[0018] Step S15: Assign the time and value fields in the data anomaly temporary table to empty.

[0019] According to the method of the first aspect of the present invention, in step S1, the method for calculating the mean value includes: the mean value includes the mean value of the preset incremental data and the mean value of the preset random data;

[0020] For data classified as preset increments,

[0021] Where AVG(·) is the mean calculation function, N is the number of data in the indicator name key field of the retrieved indicator name data, Xi is the data in the indicator name key field of the data corresponding to the current time Ti, and X(i-1) is the data in the indicator name key field of the data corresponding to the current time T(i-1);

[0022] For data classified as preset random data,

[0023] According to the method of the first aspect of the present invention, in step S1, the method for calculating the standard deviation includes: the standard deviation includes the standard deviation of the preset incremental data and the standard deviation of the preset random data;

[0024] For data classified as preset increments,

[0025]

[0026] Where STDEVP(·) is the standard deviation calculation function;

[0027] For data classified as preset random data,

[0028] According to the method of the first aspect of the present invention, in step S3, the method of querying the data anomaly temporary table based on the attribute ID and the indicator name Key of the data in the received real-time time series data, determining the classification of the data in the queried data anomaly temporary table, and calculating the data anomaly identification statistic Z based on the classification by applying the Value of the real-time time series data, the Value of the data anomaly temporary table, the time of the real-time time series data, the time, mean and standard deviation of the data anomaly temporary table includes:

[0029] Step S31: query the data anomaly temporary table based on the attribute ID and the index name Key of the data in the received real-time time series data;

[0030] Step S32: Determine whether the Value field of the data anomaly temporary table and the time field of the data anomaly temporary table are empty; if they are empty, update the time and Value of the received real-time time series data to the data anomaly temporary table;

[0031] Step S33: If the Value field of the data anomaly temporary table and the time field of the data anomaly temporary table are not empty, determine the classification of the data in the queried data anomaly temporary table;

[0032] Step S34: According to the classification, the data anomaly identification statistics include a data anomaly identification statistic Z1 of the preset incremental data and a data anomaly identification statistic Z2 of the preset random data;

[0033] If the classification is preset incremental data: data anomaly identification statistics of preset incremental data

[0034]

[0035]

[0036] If the classification is preset random data: Data anomaly identification statistics of preset random data

[0037]

[0038] Where abs(·) is the absolute value calculation function.

[0039] According to the method of the first aspect of the present invention, in step S4, if the data anomaly identification statistic Z is greater than a preset threshold, the real-time time series data is abnormal data; otherwise, the method of determining that the real-time time series data is normal data includes:

[0040] If the data anomaly identification statistic Z1 of the preset incremental data or the data anomaly identification statistic Z2 of the preset random data is greater than a preset threshold, the real-time time series data is abnormal data; otherwise, the real-time time series data is normal data;

[0041] The preset threshold is equal to 3.

[0042] A second aspect of the present invention discloses a real-time cleaning system for power time series data, the system comprising:

[0043] The first processing module is configured to obtain the attribute ID and the indicator name key of the pre-received time series data, classify the indicator name key of each data into preset incremental data or preset random data, use the attribute ID and the indicator name key of the data as query conditions, retrieve the power history data, and construct a temporary data anomaly table including: attribute ID, indicator name key of the data, classification, mean, standard deviation, time and value;

[0044] The second processing module is configured to receive real-time time series data with a data structure of attribute ID, time, data indicator name Key and Value;

[0045] The third processing module is configured to query the data anomaly temporary table according to the attribute ID and the indicator name Key of the data in the received real-time time series data, determine the classification of the data in the queried data anomaly temporary table, and calculate the data anomaly identification statistic Z based on the classification by applying the Value of the real-time time series data, the Value of the data anomaly temporary table, the time of the real-time time series data, the time of the data anomaly temporary table, the mean and the standard deviation;

[0046] The fourth processing module is configured to: if the data anomaly identification statistic Z is greater than a preset threshold, the real-time time series data is abnormal data; otherwise, the real-time time series data is normal data.

[0047] According to the system of the second aspect of the present invention, the first processing module is configured to classify the indicator name Key of each time series data into preset incremental data or preset random data, including:

[0048] If the data value continues to increase with time until it reaches the maximum value of the data and then increases again from 0, the data is defined as preset incremental data;

[0049] If the data values ​​do not have monotonicity as time increases and appear to be disordered, the data is defined as preset random data.

[0050] According to the system of the second aspect of the present invention, the first processing module is configured to construct a temporary data anomaly table including attribute ID, data indicator name Key, classification, mean, standard deviation, time and Value, including:

[0051] Get the attribute ID and the indicator name of the pre-received time series data.

[0052] Classify the indicator name Key of each data into preset incremental data or preset random data;

[0053] Use the attribute ID and the data indicator name Key as the query conditions to retrieve one year's electricity historical data;

[0054] Calculate the mean and standard deviation based on the preset incremental data or the data type of the preset random data of the indicator name Key of each data of the pre-received time series data, and fill in the mean and standard deviation fields in the data anomaly temporary table accordingly;

[0055] Assign the time and value fields in the data exception temporary table to empty.

[0056] According to the system of the second aspect of the present invention, the first processing module is configured such that the calculation of the mean value includes: the mean value includes the mean value of the preset incremental data and the mean value of the preset random data;

[0057] For data classified as preset increments,

[0058] Where AVG(·) is the mean calculation function, N is the number of data in the indicator name key field of the retrieved indicator name data, Xi is the data in the indicator name key field of the data corresponding to the current time Ti, and X(i-1) is the data in the indicator name key field of the data corresponding to the current time T(i-1);

[0059] For data classified as preset random data,

[0060] According to the system of the second aspect of the present invention, the first processing module is configured such that the calculation of the standard deviation includes: the standard deviation includes the standard deviation of the preset incremental data and the standard deviation of the preset random data;

[0061] For data classified as preset increments,

[0062]

[0063] Where STDEVP(·) is the standard deviation calculation function;

[0064] For data classified as preset random data,

[0065] According to the system of the second aspect of the present invention, the third processing module is configured to query the data anomaly temporary table based on the attribute ID and the indicator name Key of the data in the received real-time time series data, determine the classification of the data in the queried data anomaly temporary table, and calculate the data anomaly identification statistic Z based on the classification by applying the Value of the real-time time series data, the Value of the data anomaly temporary table, the time of the real-time time series data, the time of the data anomaly temporary table, the mean and the standard deviation, including:

[0066] Query the data anomaly temporary table based on the attribute ID and the indicator name Key in the received real-time time series data;

[0067] Determine whether the Value field of the data anomaly temporary table and the time field of the data anomaly temporary table are empty; if they are empty, update the time and Value of the received real-time time series data to the data anomaly temporary table;

[0068] If the Value field of the data anomaly temporary table and the time field of the data anomaly temporary table are not empty, determine the classification of the data in the queried data anomaly temporary table;

[0069] According to the classification, the data anomaly identification statistics include data anomaly identification statistics Z1 of preset incremental data and data anomaly identification statistics Z2 of preset random data;

[0070] If the classification is preset incremental data: data anomaly identification statistics of preset incremental data

[0071]

[0072]

[0073] If the classification is preset random data: Data anomaly identification statistics of preset random data

[0074]

[0075] Where abs(·) is the absolute value calculation function.

[0076] According to the system of the second aspect of the present invention, the fourth processing module is configured to: if the data anomaly identification statistic Z is greater than a preset threshold, the real-time time series data is abnormal data; otherwise, the real-time time series data is normal data, including:

[0077] If the data anomaly identification statistic Z1 of the preset incremental data or the data anomaly identification statistic Z2 of the preset random data is greater than a preset threshold, the real-time time series data is abnormal data; otherwise, the real-time time series data is normal data;

[0078] The preset threshold is equal to 3.

[0079] A third aspect of the present invention discloses an electronic device comprising a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of any one of the methods for real-time cleaning of power time series data according to the first aspect of the present disclosure.

[0080] A fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods for real-time cleaning of power time series data according to the first aspect of the present disclosure.

[0081] The solution proposed by the present invention, by constructing a temporary table for data anomaly judgment and dynamically updating it, realizes that the anomaly cleaning of real-time data only relies on the temporary table and the previous data, saving real-time computing resources and meeting the performance requirements under high-speed data flow. At the same time, the time series data is divided into preset incremental data and preset random data, and the corresponding mean and standard deviation of the temporary table are calculated separately. A normal distribution model is adopted, and the Z value is used to judge the anomaly of different types of time series data, avoiding the uncertainty of using limit values ​​to judge the anomaly of data values ​​(such as: preset incremental data, negative burrs appear, and the anomaly cannot be determined by the limit value; the limit value of preset random data is not easy to determine). BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0083] Figure 1 This is a flow chart of a method for real-time cleaning of power time series data according to an embodiment of the present invention;

[0084] Figure 2 This is an example of preset incremental data according to an embodiment of the present invention;

[0085] Figure 3 An example of preset random data according to an embodiment of the present invention;

[0086] Figure 4 This is a structural diagram of a real-time cleaning system for power time series data according to an embodiment of the present invention;

[0087] Figure 5 FIG. 4 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0088] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0089] A first aspect of the present invention discloses a real-time cleaning method for power time series data. Figure 1 FIG. 1 is a flow chart of a method for real-time cleaning of power time series data according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0090] Step S1: Obtain the attribute ID and the index name key of the pre-received time series data, classify the index name key of each data into preset incremental data or preset random data, use the attribute ID and the index name key of the data as query conditions, retrieve the power history data, and construct a temporary data anomaly table including: attribute ID, index name key of the data, classification, mean, standard deviation, time and Value;

[0091] Step S2: Receive real-time time series data with a data structure consisting of attribute ID, time, data indicator name Key and Value in real time;

[0092] Step S3: query the data anomaly temporary table based on the attribute ID and the indicator name Key of the data in the received real-time time series data, determine the classification of the data in the queried data anomaly temporary table, and calculate the data anomaly identification statistic Z based on the classification by applying the Value of the real-time time series data, the Value of the data anomaly temporary table, the time of the real-time time series data, the time of the data anomaly temporary table, the mean and the standard deviation;

[0093] Step S4: If the data anomaly identification statistic Z is greater than a preset threshold, the real-time time series data is abnormal data; otherwise, the real-time time series data is normal data.

[0094] In step S1, the attribute ID of the pre-received time series data and the indicator name Key of the data are obtained, and the indicator name Key of each data is classified into preset incremental data or preset random data. The attribute ID and the indicator name Key of the data are used as query conditions to retrieve the power history data and construct a data anomaly temporary table including: attribute ID, indicator name Key of the data, classification, mean, standard deviation, time and Value.

[0095] In some embodiments, in step S1, the method of classifying the indicator name Key of each data into preset incremental data or preset random data includes:

[0096] like Figure 2 As shown, if the data value continues to increase with time until it reaches the maximum value of the data and then increases again from 0, the data is defined as preset incremental data, such as active power, reactive power, operating time, etc.;

[0097] like Figure 3 As shown in FIG, if the data value does not have monotonicity as time increases and appears to be in a disordered state, the data is defined as preset random data.

[0098] The preset random data is divided into random data within a preset range and random data of device operation;

[0099] Random data within the preset range changes within a certain range of its effective value, including voltage, power factor, harmonics, frequency, etc.

[0100] The valid range of random data of equipment operation is from 0 to the maximum value. Its value is related to the power generation / consumption of the equipment, including current, power, etc.

[0101] The method for constructing a temporary data anomaly table including attribute ID, data indicator name Key, classification, mean, standard deviation, time and Value includes:

[0102] Step S11: Obtain the attribute ID of the pre-received time series data and the index name of the data, i.e., the index name Key;

[0103] Step S12: classify the indicator name Key of each data into preset incremental data or preset random data;

[0104] Step S13: Using the attribute ID and the data indicator name Key as query conditions, retrieve one year's electricity history data;

[0105] Step S14: Calculate the mean and standard deviation based on the preset incremental data or the data type of the preset random data of the indicator name Key of each data of the pre-received time series data, and fill them into the mean and standard deviation fields in the data anomaly temporary table accordingly;

[0106] Step S15: Assign the time and value fields in the data anomaly temporary table to empty.

[0107] As shown in Table 1, the temporary table structure is: attribute, key, type, time, value, mean, and standard deviation.

[0108] Table 1

[0109]

[0110] Attribute: refers to the source of data or device information, such as device number or device name, Id;

[0111] Key: The name of the indicator of the data sampled by the IoT device, which is known information;

[0112] Category: includes two types: preset incremental data and preset random data.

[0113] Time: indicates the time of the previous data of the corresponding attribute and Key field, represented by T(i-1);

[0114] Value: indicates the value of the previous data of the corresponding attribute and Key field, represented by X(i-1);

[0115] Mean: represents the average value of one year's data for the corresponding attribute or key field. The value is updated daily or monthly.

[0116] Standard Deviation: Indicates the standard deviation of the corresponding attribute, Key field, and one year of data. The value is updated daily or monthly.

[0117] The method for calculating the mean value includes: the mean value includes the mean value of preset incremental data and the mean value of preset random data;

[0118] For data classified as preset increments,

[0119] Where AVG(·) is the mean calculation function, N is the number of data in the indicator name key field of the retrieved indicator name data, Xi is the data in the indicator name key field of the data corresponding to the current time Ti, and X(i-1) is the data in the indicator name key field of the data corresponding to the current time T(i-1);

[0120] For data classified as preset random data,

[0121] The method for calculating the standard deviation includes: the standard deviation includes the standard deviation of the preset incremental data and the standard deviation of the preset random data;

[0122] For data classified as preset increments,

[0123]

[0124] Where STDEVP(·) is the standard deviation calculation function;

[0125] For data classified as preset random data,

[0126] In step S2, real-time time series data with a data structure consisting of attribute Id, time, data indicator name Key and Value is received in real time.

[0127] Specifically, after the message middleware Kafka receives real-time time series data, the Flink component obtains the data in real time, including the real-time time series data of the data structure attributes Id, time, data indicator name Key and Value.

[0128] Attribute: refers to the source of data or device information, such as device number or device name, represented by Id, which is known information;

[0129] Time: the specific time attribute of the data;

[0130] Key: The name of the indicator of the data sampled by the IoT device, which is known information; such as active power, reactive power, operating time, voltage, current, power, harmonics, etc.

[0131] Value: The specific numerical value of the field corresponding to the Key.

[0132] In step S3, according to the attribute Id in the received real-time time series data and the indicator name Key of the data, the data anomaly temporary table is queried to determine the classification of the data in the queried data anomaly temporary table. According to the classification, the Value value of the real-time time series data, the Value value of the data anomaly temporary table, the time of the real-time time series data, the time, mean and standard deviation of the data anomaly temporary table are applied to calculate the data anomaly identification statistic Z.

[0133] In some embodiments, in step S3, the method of querying the data anomaly temporary table based on the attribute ID and the indicator name Key of the data in the received real-time time series data, determining the classification of the data in the queried data anomaly temporary table, and calculating the data anomaly identification statistic Z based on the classification by applying the Value of the real-time time series data, the Value of the data anomaly temporary table, the time of the real-time time series data, the time, mean, and standard deviation of the data anomaly temporary table includes:

[0134] Step S31: query the data anomaly temporary table based on the attribute ID and the index name Key of the data in the received real-time time series data;

[0135] Step S32: Determine whether the Value field of the data anomaly temporary table and the time field of the data anomaly temporary table are empty. If they are empty, update the time and Value of the received real-time time series data to the data anomaly temporary table.

[0136] Step S33: If the Value field of the data anomaly temporary table and the time field of the data anomaly temporary table are not empty, determine the classification of the data in the queried data anomaly temporary table;

[0137] Step S34: According to the classification, the data anomaly identification statistics include a data anomaly identification statistic Z1 of the preset incremental data and a data anomaly identification statistic Z2 of the preset random data;

[0138] If the classification is preset incremental data: data anomaly identification statistics of preset incremental data

[0139]

[0140]

[0141] If the classification is preset random data: Data anomaly identification statistics of preset random data

[0142]

[0143] Where abs(·) is the absolute value calculation function.

[0144] In step S4, if the data anomaly identification statistic Z is greater than a preset threshold, the real-time time series data is abnormal data; otherwise, the real-time time series data is normal data.

[0145] In some embodiments, in step S4, if the data anomaly identification statistic Z is greater than a preset threshold, the real-time time series data is abnormal data; otherwise, the method for determining that the real-time time series data is normal data includes:

[0146] If the data anomaly identification statistic Z1 of the preset incremental data or the data anomaly identification statistic Z2 of the preset random data is greater than a preset threshold, the real-time time series data is abnormal data; otherwise, the real-time time series data is normal data;

[0147] The preset threshold is equal to 3.

[0148] Specifically, abnormal data include sudden changes in values ​​and missing values.

[0149] The abnormal data of the preset incremental data refers to the positive and reverse excessive mutations and missing values ​​of the data;

[0150] The anomaly of preset random data refers to the sudden change of the value beyond the limit.

[0151] In some embodiments, step S5 updates the time and value fields in the data anomaly temporary table and assigns them to the time and value in the real-time time series data.

[0152] In some embodiments, step S6, the normal real-time time series data is sent to a subsequent program for real-time calculation or stored in a time series database.

[0153] In summary, the solution proposed in the present invention can construct a temporary table for judging data anomalies and dynamically update it, so that the anomaly cleaning of real-time data only depends on the temporary table and the previous data, saving real-time computing resources and meeting the performance requirements under high-speed data flow. At the same time, the time series data is divided into preset incremental data and preset random data, and the corresponding mean and standard deviation of the temporary table are calculated respectively. A normal distribution model is adopted, and the Z value is used to judge the anomaly of different types of time series data, avoiding the uncertainty of using limit values ​​to judge the anomaly of data values ​​(such as: preset incremental data, negative burrs appear, and the anomaly cannot be obtained through the limit value; the limit value of preset random data is not easy to determine).

[0154] A second aspect of the present invention discloses a real-time cleaning system for power time series data. Figure 4 FIG. 1 is a structural diagram of a real-time cleaning system for power time series data according to an embodiment of the present invention; FIG. Figure 4 As shown, the system 100 includes:

[0155] The first processing module 101 is configured to obtain the attribute ID and the index name key of the pre-received time series data, classify the index name key of each data into preset incremental data or preset random data, use the attribute ID and the index name key of the data as query conditions, retrieve the power history data, and construct a temporary data anomaly table including the attribute ID, the index name key of the data, the classification, the mean, the standard deviation, the time, and the value;

[0156] The second processing module 102 is configured to receive real-time time series data with a data structure of attribute ID, time, data indicator name Key and Value;

[0157] The third processing module 103 is configured to query the data anomaly temporary table according to the attribute ID and the indicator name Key of the data in the received real-time time series data, determine the classification of the data in the queried data anomaly temporary table, and calculate the data anomaly identification statistic Z based on the classification by applying the Value of the real-time time series data, the Value of the data anomaly temporary table, the time of the real-time time series data, the time of the data anomaly temporary table, the mean and the standard deviation;

[0158] The fourth processing module 104 is configured to: if the data anomaly identification statistic Z is greater than a preset threshold, the real-time time series data is abnormal data; otherwise, the real-time time series data is normal data.

[0159] According to the system of the second aspect of the present invention, the first processing module 101 is configured to classify the indicator name Key of each time series data into preset incremental data or preset random data, including:

[0160] If the data value continues to increase with time until it reaches the maximum value of the data and then increases again from 0, the data is defined as preset incremental data;

[0161] If the data values ​​do not have monotonicity as time increases and appear to be disordered, the data is defined as preset random data.

[0162] According to the system of the second aspect of the present invention, the first processing module 101 is configured to construct a temporary data anomaly table including attribute ID, data indicator name Key, classification, mean, standard deviation, time, and Value, including:

[0163] Get the attribute ID and the indicator name of the pre-received time series data.

[0164] Classify the indicator name Key of each data into preset incremental data or preset random data;

[0165] Use the attribute ID and the data indicator name Key as the query conditions to retrieve one year's worth of historical electricity data.

[0166] Calculate the mean and standard deviation based on the preset incremental data or the data type of the preset random data of the indicator name Key of each data of the pre-received time series data, and fill in the mean and standard deviation fields in the data anomaly temporary table accordingly;

[0167] Assign the time and value fields in the data exception temporary table to empty.

[0168] According to the system of the second aspect of the present invention, the first processing module 101 is configured such that the calculation of the mean value includes: the mean value includes the mean value of the preset incremental data and the mean value of the preset random data;

[0169] For data classified as preset increments,

[0170] Where AVG(·) is the mean calculation function, N is the number of data in the indicator name key field of the retrieved indicator name data, Xi is the data in the indicator name key field of the data corresponding to the current time Ti, and X(i-1) is the data in the indicator name key field of the data corresponding to the current time T(i-1);

[0171] For data classified as preset random data,

[0172] According to the system of the second aspect of the present invention, the first processing module 101 is configured such that the calculation of the standard deviation includes: the standard deviation includes the standard deviation of the preset incremental data and the standard deviation of the preset random data;

[0173] For data classified as preset increments,

[0174]

[0175] Where STDEVP(·) is the standard deviation calculation function;

[0176] For data classified as preset random data,

[0177] According to the system of the second aspect of the present invention, the third processing module 103 is configured to query the data anomaly temporary table based on the attribute ID and the indicator name Key of the data in the received real-time time series data, determine the classification of the data in the queried data anomaly temporary table, and calculate the data anomaly identification statistic Z based on the classification by applying the Value of the real-time time series data, the Value of the data anomaly temporary table, the time of the real-time time series data, the time of the data anomaly temporary table, the mean and the standard deviation, including:

[0178] Query the data anomaly temporary table based on the attribute ID and the indicator name Key in the received real-time time series data;

[0179] Determine whether the Value field of the data anomaly temporary table and the time field of the data anomaly temporary table are empty; if they are empty, update the time and Value of the received real-time time series data to the data anomaly temporary table;

[0180] If the Value field of the data anomaly temporary table and the time field of the data anomaly temporary table are not empty, determine the classification of the data in the queried data anomaly temporary table;

[0181] According to the classification, the data anomaly identification statistics include a data anomaly identification statistic Z1 of preset incremental data and a data anomaly identification statistic Z2 of preset random data;

[0182] If the classification is preset incremental data: data anomaly identification statistics of preset incremental data

[0183]

[0184]

[0185] If the classification is preset random data: Data anomaly identification statistics of preset random data

[0186]

[0187] Where abs(·) is the absolute value calculation function.

[0188] According to the system of the second aspect of the present invention, the fourth processing module 104 is configured to: if the data anomaly identification statistic Z is greater than a preset threshold, the real-time time series data is abnormal data; otherwise, the real-time time series data is normal data, including:

[0189] If the data anomaly identification statistic Z1 of the preset incremental data or the data anomaly identification statistic Z2 of the preset random data is greater than a preset threshold, the real-time time series data is abnormal data; otherwise, the real-time time series data is normal data;

[0190] The preset threshold is equal to 3.

[0191] A third aspect of the present invention discloses an electronic device comprising a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of any one of the methods for real-time cleaning of power time series data disclosed in the first aspect of the present invention.

[0192] Figure 5 FIG. 1 is a structural diagram of an electronic device according to an embodiment of the present invention. Figure 5 As shown, the electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the electronic device housing, or an external keyboard, touchpad or mouse.

[0193] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a structural diagram of the part related to the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0194] A fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any one of the steps in the method for real-time cleaning of power time series data disclosed in the first aspect of the present invention.

[0195] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above embodiments only express several implementation methods of the present application. The description is relatively specific and detailed, but it cannot be understood as a limitation on the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of this application, several variations and improvements can be made, which all fall within the scope of protection of this application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.

Claims

1. A real-time cleaning method for power time series data, characterized in that: The method comprises: Step S1: Obtain the attribute ID of the pre-received time series data and the indicator name Key of the time series data, classify the preset incremental data or preset random data for the indicator name Key of each time series data, use the attribute ID and the indicator name Key of the data as query conditions, retrieve the power history data, and construct a data anomaly temporary table including: attribute ID, indicator name Key of the data, classification, mean, standard deviation, time and Value; Step S2: Receive real-time time series data with a data structure of attribute ID, time and data indicator name Key and Value; Step S3: query the data anomaly temporary table based on the attribute ID and the indicator name Key of the data in the received real-time time series data, determine the classification of the data in the queried data anomaly temporary table, and calculate the data anomaly identification statistic Z based on the classification by applying the Value of the real-time time series data, the Value of the data anomaly temporary table, the time of the real-time time series data, the time of the data anomaly temporary table, the mean and the standard deviation; Step S4: If the data anomaly identification statistic Z is greater than a preset threshold, the real-time time series data is abnormal data; otherwise, the real-time time series data is normal data.

2. A method for real-time cleaning of power time series data according to claim 1, characterized in that: In step S1, the method of classifying the indicator name Key of each time series data into preset incremental data or preset random data includes: If the data value continues to increase with time until it reaches the maximum value of the data and then increases again from 0, the data is defined as preset incremental data; If the data values ​​do not have monotonicity as time increases and appear to be disordered, the data is defined as preset random data.

3. A method for real-time cleaning of power time series data according to claim 2, characterized in that: In step S1, the method of constructing a temporary data anomaly table including attribute ID, data indicator name Key, classification, mean, standard deviation, time and Value includes: Step S11: Obtain the attribute ID of the pre-received time series data and the index name of the data, i.e., the index name Key; Step S12: classify the indicator name Key of each data into preset incremental data or preset random data; Step S13: Using the attribute ID and the data indicator name Key as query conditions, retrieve one year's electricity history data; Step S14: Calculate the mean and standard deviation based on the preset incremental data or the data type of the preset random data of the indicator name Key of each data of the pre-received time series data, and fill them into the mean and standard deviation fields in the data anomaly temporary table accordingly; Step S15: Assign the time and value fields in the data anomaly temporary table to empty.

4. A method for real-time cleaning of power time series data according to claim 3, characterized in that: In the step S1, the method for calculating the mean includes: the mean includes the mean of preset incremental data and the mean of preset random data; For data classified as preset increments, Where AVG(·) is the mean calculation function, N is the number of data in the indicator name key field of the retrieved indicator name data, Xi is the data in the indicator name key field of the data corresponding to the current time Ti, and X(i-1) is the data in the indicator name key field of the data corresponding to the current time T(i-1); For data classified as preset random data, 5. The method for real-time cleaning of power time series data according to claim 3, characterized in that: In the step S1, the method for calculating the standard deviation includes: the standard deviation includes the standard deviation of the preset incremental data and the standard deviation of the preset random data; For data classified as preset increments, Where STDEVP(·) is the standard deviation calculation function; For data classified as preset random data, 6. The method for real-time cleaning of power time series data according to claim 1, characterized in that: In step S3, the method of querying the data anomaly temporary table based on the attribute ID and the indicator name Key of the data in the received real-time time series data, determining the classification of the data in the queried data anomaly temporary table, and calculating the data anomaly identification statistic Z based on the classification by applying the Value of the real-time time series data, the Value of the data anomaly temporary table, the time of the real-time time series data, the time, mean and standard deviation of the data anomaly temporary table includes: Step S31: query the data anomaly temporary table based on the attribute ID and the index name Key of the data in the received real-time time series data; Step S32: Determine whether the Value field of the data anomaly temporary table and the time field of the data anomaly temporary table are empty; if they are empty, update the time and Value of the received real-time time series data to the data anomaly temporary table; Step S33: If the Value field of the temporary data anomaly table and the time field of the temporary data anomaly table are not empty, determine the classification of the data in the queried temporary data anomaly table; Step S34: According to the classification, the data anomaly identification statistics include a data anomaly identification statistic Z1 of the preset incremental data and a data anomaly identification statistic Z2 of the preset random data; If the classification is preset incremental data: data anomaly identification statistics of preset incremental data If the classification is preset random data: Data anomaly identification statistics of preset random data Where abs(·) is the absolute value calculation function.

7. A method for real-time cleaning of power time series data according to claim 6, characterized in that: In step S4, if the data anomaly identification statistic Z is greater than a preset threshold, the real-time time series data is abnormal data; otherwise, the real-time time series data is normal data. The method includes: If the data anomaly identification statistic Z1 of the preset incremental data or the data anomaly identification statistic Z2 of the preset random data is greater than a preset threshold, the real-time time series data is abnormal data; otherwise, the real-time time series data is normal data; The preset threshold is equal to 3.

8. A real-time cleaning system for power time series data, characterized in that: The system comprises: The first processing module is configured to obtain the attribute ID and the indicator name key of the pre-received time series data, classify the indicator name key of each data into preset incremental data or preset random data, use the attribute ID and the indicator name key of the data as query conditions, retrieve the power history data, and construct a temporary data anomaly table including: attribute ID, indicator name key of the data, classification, mean, standard deviation, time and value; The second processing module is configured to receive real-time time series data with a data structure of attribute ID, time, data indicator name Key and Value; The third processing module is configured to query the data anomaly temporary table according to the attribute ID and the indicator name Key of the data in the received real-time time series data, determine the classification of the data in the queried data anomaly temporary table, and calculate the data anomaly identification statistic Z based on the classification by applying the Value of the real-time time series data, the Value of the data anomaly temporary table, the time of the real-time time series data, the time of the data anomaly temporary table, the mean and the standard deviation; The fourth processing module is configured to: if the data anomaly identification statistic Z is greater than a preset threshold, the real-time time series data is abnormal data; otherwise, the real-time time series data is normal data.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps of the real-time cleaning method for power time series data described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the method for real-time cleaning of power time series data according to any one of claims 1 to 7 are implemented.

Images