Multi-source heterogeneous abnormal data cleaning method for power monitoring system

Abstract

The invention discloses a multi-source heterogeneous abnormal data cleaning method for an electric power monitoring system, and belongs to the technical field of data processing of the electric power monitoring system. The invention aims to identify, process and clean abnormal data from power monitoring data of different sources and heterogeneous formats. The method comprises the following steps: firstly collecting and arranging related attribute information of data, including data sources, data types and timestamps, to form a preliminary data set, processing the collected multi-source heterogeneous data, identifying and marking abnormal data in the data, and performing targeted cleaning to improve the data quality; and data with high accuracy, good integrity and low redundancy are provided for the power monitoring system. According to the method, the data redundancy is effectively reduced, the data quality is remarkably improved, and a solid data basis is provided for stable operation and efficient decision making of a power monitoring system.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing of electric power monitoring systems, and in particular relates to a method for cleaning multi-source heterogeneous abnormal data of electric power monitoring systems. Background Art

[0002] With the rapid development of Internet technology, the amount of data has exploded. In order to effectively cope with the needs of large-scale data storage and access, different types of databases have been developed, resulting in a large amount of multi-source heterogeneous data. In the process of processing multi-source heterogeneous data in the database, due to external environmental interference, data transmission errors and other reasons, there may be some abnormal data in the database, which will have a certain impact on the use of database data, resulting in reduced database application efficiency.

[0003] The power monitoring system plays an important role in real-time monitoring of the operating status of power equipment, detecting power quality, and analyzing power consumption behavior. However, the power monitoring system often faces the challenge of multi-source heterogeneous data. These data may come from equipment from different manufacturers, different sensors, etc., with different data formats, accuracy, sampling frequencies, etc. This multi-source heterogeneous data brings certain difficulties to the data cleaning work of the power monitoring system. At present, there are mainly the following problems in cleaning the multi-source heterogeneous abnormal data of the power monitoring system:

[0004] (1) In conventional databases, the cleaning of multi-source heterogeneous abnormal data is mainly carried out by calculating the similarity value of data features, which ignores the impact of the temporal correlation of data features on the identification of abnormal data, resulting in a low recall rate of the abnormal data cleaning results;

[0005] (2) Traditional data cleaning methods are often designed for a single data source and are difficult to effectively process multi-source heterogeneous data.

[0006] In view of the multi-source heterogeneous data characteristics of the power monitoring system, it is necessary to be compatible with the data formats of different data sources, automatically identify abnormal data and perform cleaning processing to ensure data quality and accuracy, and provide a reliable data foundation for subsequent data analysis and application. Therefore, a new technical solution is urgently needed in the existing technology to solve the above problems. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a method for cleaning multi-source heterogeneous abnormal data of an electric power monitoring system to solve the technical problems that traditional data cleaning methods are often designed for a single data source and are difficult to effectively process multi-source heterogeneous data; and the recall rate of the multi-source heterogeneous abnormal data cleaning results using the method of calculating the data feature similarity value is low.

[0008] A method for cleaning multi-source heterogeneous abnormal data of a power monitoring system comprises the following steps, which are performed in sequence:

[0009] Step 1: Collect and represent multi-source heterogeneous data of the power monitoring system;

[0010] The devices from which the multi-source heterogeneous data of the power monitoring system comes are numbered, and the multi-source heterogeneous data are collected at set time intervals every day. The data are collected for multiple days and stored. The multi-source heterogeneous data are divided into four categories: device parameters, market parameters, environmental parameters, and state parameters. Each type of parameter includes specific sub-parameters.

[0011] Step 2: Preprocess and convert the multi-source heterogeneous data of the power monitoring system into a unified format;

[0012] Step 3: Determine whether the multi-source heterogeneous data of the power monitoring system is abnormal data, and clean the abnormal data;

[0013] The abnormal data includes duplicate data, missing data and erroneous data;

[0014] The cleaning includes deduplication of duplicate data, supplementation of missing data or abnormal alarm when there is too much missing data, judgment of erroneous data and data correction.

[0015] The format in step 2 is a timestamp plus a value. The timestamp T of the i-th device at the t-th time is t i Represented as T t i :[YYYY-MM-DD:t], where YYYY is the four-digit year, MM is the two-digit month, and DD is the two-digit day.

[0016] The deduplication of duplicate data in step 3 includes the following steps:

[0017] The multi-source heterogeneous data of the i-th device at time t is expressed by the following sub-parameters: Device voltage Active power of the device Reactive power of the equipment Power usage of the device Real-time market electricity price RTP t 、Historical electricity price HEP t , Ambient temperature Ambient humidity The operating status of the device The collected multi-source heterogeneous data of the power monitoring system are compared with the RTP t 、HEPt , timestamp and value to identify whether there is duplicate data;

[0018] First, according to the timestamp T t i Find the current of all devices i at the same time t Device voltage Active power of the device Reactive power of the equipment Power usage of the device Real-time market electricity price RTP t 、Historical electricity price HEP t , Ambient temperature Ambient humidity use Indicates the data value collected by variable S at the tth time of the i-th device. RTP t 、HEP t ,

[0019] Set the device standard value to SV t s , the allowable error of the equipment is EV t s , then define:

[0020] EV t s =SV t s ×5% (1);

[0021] Defining DV t i [S] is the data value collected by variable S at the tth time of the i-th device The standard value of the device is SV t s The absolute value of the difference is expressed as:

[0022]

[0023] Then This achieves the effect of data deduplication;

[0024] The operating status of the i-th device at time t Divided into X 1 ,X 2 ,X 3 Three parts, with X j Indicates that j = 1, 2, 3, the operating status of the i-th device at time t′ Divided into X′ 1,X′ 2 ,X′ 3 Three parts, with X′ j Indicates, j = 1, 2, 3;

[0025] Define the running status similarity function The operating status of the i-th device at time t and the operating status of the i-th device at time t′ For comparison:

[0026]

[0027] in, Represents XOR operation;

[0028] X j With X′ j Same, then

[0029] X j With X′ j Different, then

[0030] Define the running status similarity judgment index MatOS:

[0031]

[0032] MatOS=0, which means the running status of the ith device at the tth time and the operating status of the i-th device at time t′ Duplicate, delete The value of

[0033] MatOS=1, which means the running status of the ith device at the tth time and the operating status of the i-th device at time t′ If the equipment is inconsistent, a notification is sent to the staff for secondary confirmation. If the staff confirms that the equipment is abnormal, the abnormal equipment is manually handled and the operating status value is updated. If the staff confirms that the equipment is working normally, it means that the operating status of the i-th equipment at time t and time t′ has changed, and the new status value and the corresponding value of the new status value are retained. RTP t 、HEP t ,

[0034] The steps for supplementing missing data or issuing abnormal alarms for excessive missing data in step 3 are as follows:

[0035] The time characteristic of the missing value of the i-th device is recorded as Time, and tb is the starting time of the missing time period, t f is the end time of the missing time period, and the time characteristics are divided into three categories: single time, short-term time, and long-term time:

[0036]

[0037] If Time is a single time, the mean method is used to supplement the missing data. The calculation formula is:

[0038]

[0039] Where Cde(t) represents the missing data value at time t, C(t-2), C(t-1), C(t+1), and C(t+2) represent the non-missing data values ​​at time t-2, time t-1, time t+1, and time t+2, respectively;

[0040] If Time is a short-term time, linear interpolation is used to interpolate the missing values. It is assumed that there are M missing values ​​in this short-term time, 2≤M≤180, and at t m The interpolation value at the moment is c(t m ), m=1,2,···,M, the formula is as follows:

[0041]

[0042] Among them, c(t f ) represents the end time t of the missing time period f The non-missing data value at time, c(t b ) represents the starting time t of the missing time period b The non-missing data value at ;

[0043] If Time is a long period of time, which means that there is too much data missing, it is considered as a device failure, and a device failure notification is sent to the staff. The staff will check whether there is any abnormality in the equipment and repair it in time.

[0044] The steps of judging and correcting the erroneous data in step 3 are as follows:

[0045] Current of the device Device voltage Active power of the device Reactive power of the equipment Power usage of the device Real-time market electricity price RTP t 、Historical electricity price HEP t , Ambient temperature Ambient humidity The values ​​are all greater than or equal to 0. If there is data less than 0, the data is considered to be wrong. The operating status value of the device Only six values ​​can be selected: 000, 001, 010, 011, 100, 101. If any other value appears, the data is considered to be wrong.

[0046] According to the timestamp T of the i-th device at time t t i Find the data value of variable S of the same device at time t within 15 days before and after the time t to be judged, and form a data set S respectively RTP t 、HEP t , time=1,2,···,30, where the day before the one to be judged is numbered as 15, and the numbering continues until it reaches 1. The day after the one to be judged is numbered as 16, and the numbering continues until it reaches 30. Calculate the data set The mean μ i {S}:

[0047]

[0048] Computational Dataset The standard deviation σ i {S}:

[0049]

[0050] use Measure whether the data value at time t is wrong, It is expressed as:

[0051]

[0052] The data value at time t is judged to be normal, otherwise, the data value at time t is judged to be wrong;

[0053] If the data is judged to be erroneous, the staff will first check whether the data error is caused by equipment abnormality. If it is caused by equipment abnormality, it will be repaired in time. Otherwise, the erroneous data will be corrected manually.

[0054] Through the above design scheme, the present invention can bring the following beneficial effects:

[0055] The present invention aims to identify, process and clean abnormal data in power monitoring data from different sources and heterogeneous formats. First, collect and organize the relevant attribute information of these data, including data source, data type, and timestamp, to form a preliminary data set. By processing the collected multi-source heterogeneous data, identify and mark the abnormal data therein, and perform targeted cleaning to improve data quality, and provide the power monitoring system with data with high accuracy, good integrity and low redundancy. The present invention not only effectively reduces data redundancy, but also significantly improves data quality, providing a solid data foundation for the stable operation and efficient decision-making of the power monitoring system. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The present invention is further described below with reference to the accompanying drawings and specific embodiments:

[0057] Figure 1 The present invention is a flow chart of a method for cleaning multi-source heterogeneous abnormal data in an electric power monitoring system. DETAILED DESCRIPTION

[0058] The present invention will be further described below using the accompanying drawings and specific embodiments;

[0059] The object of the present invention is to achieve the following technical solution: a method for cleaning multi-source heterogeneous abnormal data of a power monitoring system, such as Figure 1 As shown, the specific steps are:

[0060] Step 1: Multi-source heterogeneous data collection and representation of power monitoring system;

[0061] The multi-source heterogeneous data of the power monitoring system is collected and divided into four categories: equipment parameters, market parameters, environmental parameters, and state parameters. The multi-source heterogeneous data of the power monitoring system is represented as W = {W 1 , W 2 , W 3 , W 4}, where W 1 Indicates equipment parameters, including: equipment current I, equipment voltage U, equipment active power p, equipment reactive power Q, equipment power consumption EUD; W 2 Represents market parameters, including: real-time market electricity price RTP, historical electricity price HEP; W 3 Indicates environmental parameters, including: ambient temperature Te, ambient humidity H; W 4 Indicates status parameters, including: the operating status of the device OS;

[0062] Assume that the multi-source heterogeneous data of the power monitoring system comes from n devices in total, and these n devices are numbered from 1 to n. Assume that the time interval for obtaining the multi-source heterogeneous data of the power monitoring system is 1 minute, then 1440 data needs to be obtained every day. The multi-source heterogeneous data from the i-th device at the t-th time can be expressed as: the current of the device Device voltage Active power of the device Reactive power of the equipment Power usage of the device Real-time market electricity price RTP t 、Historical electricity price HEP t , Ambient temperature Ambient humidity The operating status of the device

[0063] Step 2: Preprocessing of multi-source heterogeneous data of power monitoring system;

[0064] Pre-process the collected multi-source heterogeneous data of the power monitoring system and convert the data into a unified format and structure;

[0065] 1) Unify the unit of multi-source heterogeneous data of power monitoring system; The unit is ampere (A). The unit is Ford (V), The unit is Watt (W). The unit is var. The unit is kilowatt-hour (kWh), RTP t The unit is yuan / kilowatt-hour (yuan / kWh), HEP t The unit is yuan / kilowatt-hour (yuan / kWh), The unit is Celsius The unit is percentage (%);

[0066] 2) RTP t 、HEP t , It is expressed in the form of timestamp plus value, the timestamp of the ith device at the tth moment Expressed as Where YYYY is the four-digit year, MM is the two-digit month, and DD is the two-digit day. is the current value, is the voltage value, is the power value, is the power value, For power consumption, RTP t For electrical value, HEP tFor the value of electricity, is the temperature value, is the humidity value, the above values ​​are rounded to two decimal places. The running status value is represented by X 1 X 2 X 3 , X 1 X 2 X 3 Refers to a value consisting of three 0s or 1s;

[0067] 3) Operation status of hardware equipment in the power monitoring system Assign a value. When the operating status of the hardware device is normal, the operating status value When the operating status of the hardware device is a fault state, the operating status value When the operating status of the hardware device is in the alarm state, the operating status value When the operating status of the hardware device is maintenance status, the operating status value When the hardware device is in standby mode, the operating status value When the running status of the hardware device is shutdown, the running status value

[0068] Step 3: Cleaning of multi-source heterogeneous abnormal data in the power monitoring system;

[0069] The multi-source heterogeneous abnormal data of the power monitoring system is divided into duplicate data, missing data and error data. The purpose of abnormal data cleaning is achieved by performing numerical deduplication, missing value processing and error value processing on the multi-source heterogeneous abnormal data of the power monitoring system.

[0070] 1) Deduplication of values

[0071] The collected multi-source heterogeneous data of the power monitoring system are compared with the RTP t 、HEP t , The timestamp and value are used to identify whether there is duplicate data. First, the timestamp Find the current of all devices i at the same time t Device voltage Active power of the device Reactive power of the equipment Power usage of the device Real-time market electricity price RTP t 、Historical electricity price HEP t , Ambient temperature Ambient humidity use It represents the data value collected by variable S at the tth time of the i-th device. The variable S can be RTP t 、HEP t ,

[0072] Assume that the equipment standard value is SV t s , the allowable error of the equipment is EV t s ,definition:

[0073] EV t s =SV t s ×5% (1)

[0074] definition is the data value collected by variable S at the tth time of the i-th device The standard value of the device is SV t s The absolute value of the difference is expressed as:

[0075]

[0076] If DV t i [S]≤EV t s , then let This achieves the effect of data deduplication;

[0077] The operating status of the i-th device at time t Divided into X 1 ,X 2 ,X 3 Three parts, with X j Indicates that j = 1, 2, 3, the operating status of the i-th device at time t′ Divided into X 1 ′,X 2 ′,X 3 'Three parts, use X j ′ means j=1,2,3;

[0078] Define the running status similarity function The operating status of the i-th device at time t and the operating status of the i-th device at time t′ For comparison:

[0079]

[0080] in, Represents XOR operation, if Xj With X j ′ is the same, then If X j With X j ′ is different, then

[0081] Define the running status similarity judgment index MatOS:

[0082]

[0083] If MatOS=0, it indicates the operating status of the i-th device at time t. and the operating status of the i-th device at time t′ Duplicate, delete If MatOS=1, it indicates the running status of the i-th device at the t-th moment. and the operating status of the i-th device at time t′ If there is any inconsistency, the staff needs to confirm it again. If the equipment works abnormally, it needs to be handled in time and the operating status value needs to be updated. If the equipment works normally, it means that the operating status of the i-th equipment at time t and time t′ has changed, and the new status value needs to be retained;

[0084] 2) Missing value processing

[0085] The time characteristic of the missing value of the i-th device is recorded as Time, which can be divided into three categories: single time, short-term time, and long-term time. Let t b is the starting time of the missing time period, t f is the end time of the missing time period, then:

[0086]

[0087] When Time is a single time, the mean method is used to handle missing data, and the calculation formula is:

[0088]

[0089] Where Cde(t) represents the missing data value at time t, C(t-2), C(t-1), C(t+1), and C(t+2) represent the non-missing data values ​​at time t-2, time t-1, time t+1, and time t+2, respectively;

[0090] When Time is a short time, linear interpolation is used to interpolate the missing values. Assume that there are M missing values ​​in this short time, 2≤M≤180, and at t m The interpolation value at the moment is c(t m ), m=1,2,···,M, the formula is as follows:

[0091]

[0092] Among them, c(t f ) represents the end time t of the missing time period f The non-missing data value at time, c(t b ) represents the starting time t of the missing time period b The non-missing data value at ;

[0093] When Time is a long time, it means that there are too many missing data, which is judged as equipment failure. The staff needs to check whether there is any abnormality in the equipment and repair it in time;

[0094] 3) Error value handling

[0095] Current of the device Device voltage Active power of the device Reactive power of the equipment Power usage of the device Real-time market electricity price RTP t 、Historical electricity price HEP t , Ambient temperature Ambient humidity The values ​​are all greater than or equal to 0. If there is data less than 0, it is considered that the data is wrong. The operating status value of the device Only six values ​​can be selected: 000, 001, 010, 011, 100, 101. If any other value appears, the data is considered to be wrong.

[0096] According to the timestamp T of the i-th device at time t t i Find the data value of variable S of the same device at time t within 15 days before and after the time t to be judged, and form a data set S respectively RTP t 、HEP t , time=1,2,···,30, where the day before the one to be judged is numbered as 15, and the numbering continues until it reaches 1. The day after the one to be judged is numbered as 16, and the numbering continues until it reaches 30. Calculate the data set The mean μ i {S}:

[0097]

[0098] Computational Dataset The standard deviation σ i {S}:

[0099]

[0100] use Measure whether the data value at time t is wrong, It is expressed as:

[0101]

[0102] like The data value at time t is judged to be normal, otherwise, the data value at time t is judged to be wrong. When the data is judged to be wrong, the staff will first check whether the data error is caused by equipment abnormality. If it is caused by equipment abnormality, it will be repaired in time, otherwise, the erroneous data will be corrected according to the actual situation.

[0103] The software program involved in the present invention is compiled based on mobile communication, network and computer processing technologies, which are technologies familiar to those skilled in the art.

[0104] The specific embodiments of the present invention are merely examples for clear explanation and are not limitations on the implementation methods. For ordinary technicians in the field, other different forms of changes or modifications can be made based on the above description. There is no need to list all the implementation methods here, and the obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A method for cleaning multi-source heterogeneous abnormal data in a power monitoring system, characterized by: The process includes the following steps, which are performed in sequence: Step 1: Collect and represent multi-source heterogeneous data of the power monitoring system; The devices from which the multi-source heterogeneous data of the power monitoring system comes are numbered, and the multi-source heterogeneous data are collected at set time intervals every day. The data are collected for multiple days and stored. The multi-source heterogeneous data are divided into four categories: device parameters, market parameters, environmental parameters, and state parameters. Each type of parameter includes specific sub-parameters. Step 2: Preprocess and convert the multi-source heterogeneous data of the power monitoring system into a unified format; Step 3: Determine whether the multi-source heterogeneous data of the power monitoring system is abnormal data, and clean the abnormal data; The abnormal data includes duplicate data, missing data and erroneous data; The cleaning includes deduplication of duplicate data, supplementation of missing data or abnormal alarm when there is too much missing data, judgment of erroneous data and data correction.

2. According to claim 1, a method for cleaning multi-source heterogeneous abnormal data of a power monitoring system is characterized by: The format in step 2 is a timestamp plus a value. The timestamp T of the i-th device at the t-th time is t i Represented as T t i :[YYYY-MM-DD:t], where YYYY is the four-digit year, MM is the two-digit month, and DD is the two-digit day.

3. The method for cleaning multi-source heterogeneous abnormal data of a power monitoring system according to claim 1 is characterized by: The deduplication of duplicate data in step 3 includes the following steps: The multi-source heterogeneous data of the i-th device at time t is expressed by the following sub-parameters: Device voltage Active power of the device Reactive power of the equipment Power usage of the device Real-time market electricity price RTP t 、Historical Electricity Price HEP t , Ambient temperature Ambient humidity The operating status of the device The collected multi-source heterogeneous data of the power monitoring system are compared with the RTP t 、HEP t , timestamp and value to identify whether there is duplicate data; First, according to the timestamp T t i Find the current of all devices i at the same time t Device voltage Active power of the device Reactive power of the equipment Power usage of the device Real-time market electricity price RTP t 、Historical Electricity Price HEP t , Ambient temperature Ambient humidity use Indicates the data value collected by variable S at the tth time of the i-th device. RTP t 、HEP t , Set the device standard value to SV t s , the allowable error of the equipment is EV t s , then define: EV t s =SV t s ×5%(1); Defining DV t i [S] is the data value collected by variable S at the tth time of the i-th device The standard value of the device is SV t s The absolute value of the difference is expressed as: Then This achieves the effect of data deduplication; The operating status of the i-th device at time t It is divided into three parts: X1, X2, and X3. j Indicates that j = 1, 2, 3, the operating status of the i-th device at time t′ Divided into three parts: X1′, X2′, and X3′, with X j ′ means j=1,2,3; Define the running status similarity function The operating status of the i-th device at time t and the operating status of the i-th device at time t′ For comparison: in, Represents the exclusive OR operation; X j With X j ′ is the same, then X j With X j ′ is different, then Define the running status similarity judgment index MatOS: MatOS=0, which means the running status of the ith device at the tth time and the operating status of the i-th device at time t′ Duplicate, delete The value of MatOS=1, which means the running status of the ith device at the tth time and the operating status of the i-th device at time t′ If the equipment is inconsistent, a notification is sent to the staff for secondary confirmation. If the staff confirms that the equipment is abnormal, the abnormal equipment is manually handled and the operating status value is updated. If the staff confirms that the equipment is working normally, it means that the operating status of the i-th equipment at time t and time t′ has changed, and the new status value and the corresponding value of the new status value are retained. RTP t 、HEP t , 4. The method for cleaning multi-source heterogeneous abnormal data of a power monitoring system according to claim 1 is characterized by: The steps for supplementing missing data or issuing abnormal alarms for excessive missing data in step 3 are as follows: The time characteristic of the missing value of the i-th device is recorded as Time, and t b is the starting time of the missing time period, t f is the end time of the missing time period, and the time features are divided into three categories: single time, short-term time, and long-term time: If Time is a single time, the mean method is used to supplement the missing data. The calculation formula is: Where Cde(t) represents the missing data value at time t, C(t-2), C(t-1), C(t+1), and C(t+2) represent the non-missing data values ​​at time t-2, time t-1, time t+1, and time t+2, respectively; If Time is a short-term time, linear interpolation is used to supplement the missing values. It is assumed that there are M missing values ​​in this short-term time, 2≤M≤180, and at t m The interpolation value at the moment is c(t m ), m=1,2,···,M, the formula is as follows: Among them, c(t f ) represents the end time t of the missing time period f The non-missing data value at time, c(t b ) represents the starting time t of the missing time period b The non-missing data value at ; If Time is a long period of time, which means that there are too many missing data, it is considered as a device failure, and a device failure notification is sent to the staff. The staff will check whether there is any abnormality in the equipment and repair it in time.

5. The method for cleaning multi-source heterogeneous abnormal data of a power monitoring system according to claim 1 is characterized by: The steps of judging and correcting the erroneous data in step 3 are as follows: Current of the device Device voltage Active power of the device Reactive power of the equipment Power usage of the device Real-time market electricity price RTP t 、Historical Electricity Price HEP t , Ambient temperature Ambient humidity The values ​​are all greater than or equal to 0. If there is data less than 0, the data is considered to be wrong. The operating status value of the device Only six values ​​can be selected: 000, 001, 010, 011, 100, 101. If any other value appears, the data is considered to be wrong. According to the timestamp T of the i-th device at time t t i Find the data value of variable S of the same device at time t within 15 days before and after the time t to be judged, and form a data set S respectively RTP t 、HEP t , time=1,2,···,30, where the day before the one to be judged is numbered as 15, and the numbering continues until it reaches 1. The day after the one to be judged is numbered as 16, and the numbering continues until it reaches 30. Calculate the data set The mean μ i {S}: Computational Dataset The standard deviation σ i {S}: use Measure whether the data value at time t is wrong, It is expressed as: The data value at time t is judged to be normal, otherwise, the data value at time t is judged to be wrong; If the data is judged to be erroneous, the staff will first check whether the data error is caused by equipment abnormality. If it is caused by equipment abnormality, it will be repaired in time. Otherwise, the erroneous data will be corrected manually.

Images