Method and device for cleaning time-series industrial control protocol data

Abstract

The present application provides a method and device for cleaning time series industrial control protocol data, the method comprising: obtaining time series data to be cleaned, historical time series data from the same data source, and a maximum speed constraint; determining a current speed constraint based on the data points, maximum speed constraint, and historical time series data in the first N cleaning windows of the time series data to be cleaned; starting from the N+1th cleaning window of the time series data to be cleaned, identifying whether each data point in the time series data to be cleaned is an abnormal data point one by one based on the speed of each data point and the current speed constraint, and cleaning the identified abnormal data points. This solution determines a matching current speed constraint based on the time series data to be cleaned, and identifies and cleans abnormal data points based on whether the speed of the data point meets the current speed constraint. Even if the time series data has large fluctuations, it can accurately clean the abnormal data points therein, thereby improving the accuracy of data cleaning.

Description

Technical Field

[0001] The present invention relates to the technical field of data cleaning, and in particular to a method and device for cleaning sequential industrial control protocol data. Background Art

[0002] With the continuous development of power system informatization, big data in the power industry, represented by time series data collected by sensors (referred to as time series data), is constantly being generated. At the same time, errors are common in time series data. For example, sensors, terminal recorders, and other devices are affected by subjective and objective factors during data collection, data transmission, and data recording, resulting in data quality issues in the final data. Low-quality data can seriously impact monitoring services such as data analysis.

[0003] Therefore, it is particularly important to identify abnormal data in time series data and clean the identified abnormal data. Cleaning abnormal data means replacing abnormal data with normal data.

[0004] Existing data cleaning methods typically set a fixed threshold and then identify any value greater than the threshold as an outlier. The problem with this approach is that in time series data, normal values ​​can fluctuate significantly over time, and the fixed threshold in this approach can easily identify values ​​within the normal fluctuation range as outliers, resulting in low accuracy. Summary of the Invention

[0005] In view of the above shortcomings of the prior art, the present invention provides a method and device for cleaning time series industrial control protocol data to improve the accuracy of data cleaning of time series data.

[0006] A first aspect of the present application provides a method for cleaning time-series industrial control protocol data, comprising:

[0007] Obtaining time series data to be cleaned, historical time series data, and a maximum speed constraint; wherein the historical time series data and the time series data to be cleaned have the same data source;

[0008] Determine a current speed constraint based on data points of the first N cleaning windows in the time series data to be cleaned, the maximum speed constraint, and the historical time series data; wherein N is a preset positive integer;

[0009] Starting from the N+1th cleaning window of the time series data to be cleaned, each data point in the time series data to be cleaned is identified one by one as an abnormal data point according to the speed of each data point and the current speed constraint, and the identified abnormal data points are cleaned.

[0010] Optionally, determining the current speed constraint based on the data points of the first N cleaning windows in the time series data to be cleaned, the maximum speed constraint, and the historical time series data includes:

[0011] Calculating the velocity of each data point in the historical time series data;

[0012] Counting the average value and standard deviation of the speed of each data point in the historical time series data, and determining the initial speed constraint according to the confidence interval width corresponding to the preset target confidence level, as well as the average value and the standard deviation;

[0013] The initial speed constraint is adjusted according to the data points of the first N cleaning windows in the time series data to be cleaned and the maximum speed constraint to obtain a current speed constraint.

[0014] Optionally, when the identified abnormal data point is an isolated abnormal point, cleaning the identified abnormal data point includes:

[0015] determining a first cleaning window starting from the timestamp of the isolated outlier point;

[0016] Calculating a finite candidate set of the isolated outlier points based on other data points whose timestamps are within the first cleaning window; wherein the other data points refer to data points different from the isolated outlier points;

[0017] The isolated outlier point is replaced by the median of the finite candidate set of the isolated outlier point to complete the cleaning of the isolated outlier point.

[0018] Optionally, when the identified abnormal data points are continuous abnormal points, cleaning the identified abnormal data points includes:

[0019] Determine an abnormal sequence consisting of continuous abnormal points; wherein the abnormal sequence refers to a sequence consisting entirely of continuous abnormal points, and neither the previous data point nor the next data point of the sequence is an abnormal data point;

[0020] Calculating a change rate from a first reference point to a second reference point of the abnormal sequence; wherein the reference point is a previous data point of the abnormal sequence, and the second reference point is a subsequent data point of the abnormal sequence;

[0021] Each continuous abnormal point in the abnormal sequence is linearly cleaned using the change speed from the first reference point to the second reference point.

[0022] Optionally, also include:

[0023] After each data point is identified, if the speed of the data point or the acceleration of multiple data points before the data point meets the preset update condition, the current speed constraint is updated according to the speed of the data point and the acceleration of multiple data points before the data point.

[0024] A second aspect of the present application provides a cleaning device for time-series industrial control protocol data, comprising:

[0025] an obtaining unit, configured to obtain time series data to be cleaned, historical time series data, and a maximum speed constraint; wherein the historical time series data and the time series data to be cleaned have the same data source;

[0026] a determining unit, configured to determine a current speed constraint based on data points of the first N cleaning windows in the time series data to be cleaned, the maximum speed constraint, and historical time series data; wherein N is a preset positive integer, and the historical time series data and the time series data to be cleaned have the same data source;

[0027] The cleaning unit is used to identify whether each data point in the time series data to be cleaned is an abnormal data point one by one according to the speed of each data point and the current speed constraint, starting from the N+1th cleaning window of the time series data to be cleaned, and clean the identified abnormal data points.

[0028] Optionally, when the determining unit determines the current speed constraint based on the data points of the first N cleaning windows in the time series data to be cleaned, the maximum speed constraint, and the historical time series data, it is specifically configured to:

[0029] Calculating the velocity of each data point in the historical time series data;

[0030] Counting the average value and standard deviation of the speed of each data point in the historical time series data, and determining the initial speed constraint according to the confidence interval width corresponding to the preset target confidence level, as well as the average value and the standard deviation;

[0031] The initial speed constraint is adjusted according to the data points of the first N cleaning windows in the time series data to be cleaned and the maximum speed constraint to obtain a current speed constraint.

[0032] Optionally, when the identified abnormal data point is an isolated abnormal point, the cleaning unit is specifically configured to:

[0033] determining a first cleaning window starting from the timestamp of the isolated outlier point;

[0034] Calculating a finite candidate set of the isolated outlier points based on other data points whose timestamps are within the first cleaning window; wherein the other data points refer to data points different from the isolated outlier points;

[0035] The isolated outlier point is replaced by the median of the finite candidate set of the isolated outlier point to complete the cleaning of the isolated outlier point.

[0036] Optionally, when the identified abnormal data points are continuous abnormal points, the cleaning unit is specifically configured to:

[0037] Determine an abnormal sequence consisting of continuous abnormal points; wherein the abnormal sequence refers to a sequence consisting entirely of continuous abnormal points, and neither the previous data point nor the next data point of the sequence is an abnormal data point;

[0038] Calculating a change rate from a first reference point to a second reference point of the abnormal sequence; wherein the reference point is a previous data point of the abnormal sequence, and the second reference point is a subsequent data point of the abnormal sequence;

[0039] Each continuous abnormal point in the abnormal sequence is linearly cleaned using the change speed from the first reference point to the second reference point.

[0040] Optionally, the device further includes an updating unit, configured to:

[0041] After each data point is identified, if the speed of the data point or the acceleration of multiple data points before the data point meets the preset update condition, the current speed constraint is updated according to the speed of the data point and the acceleration of multiple data points before the data point.

[0042] The present application provides a method and device for cleaning time series industrial control protocol data, the method comprising: obtaining time series data to be cleaned, historical time series data from the same data source, and a maximum speed constraint; determining a current speed constraint based on the data points, maximum speed constraint, and historical time series data in the first N cleaning windows of the time series data to be cleaned; starting from the N+1th cleaning window of the time series data to be cleaned, identifying whether each data point in the time series data to be cleaned is an abnormal data point one by one based on the speed of each data point and the current speed constraint, and cleaning the identified abnormal data points. This solution determines a matching current speed constraint based on the time series data to be cleaned, and identifies and cleans abnormal data points based on whether the speed of the data point meets the current speed constraint. Even if the time series data has large fluctuations, it can accurately clean the abnormal data points therein, thereby improving the accuracy of data cleaning. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0044] Figure 1 A method for cleaning time-series industrial control protocol data provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of cleaning isolated abnormal points provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of cleaning continuous abnormal points provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of the structure of a cleaning device for sequential industrial control protocol data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0049] Time series data, referred to as time series data, can be understood as a collection of values ​​of at least one parameter at various time points. Time series data can usually be collected in real time using sensors.

[0050] Taking the power system as an example, a current sensor can be used to collect the current at a certain point in the power grid at every moment in real time. The collected values ​​are: 13A at 00:00:00 on July 1, 13.5A at 00:00:01 on July 1, 14A at 00:00:02 on July 1, and 20A at 00:a:b on July 1. These values ​​can be represented by time series data (13, 13.5, 14...20).

[0051] In the solution of this application, for ease of explanation, each value in the time series data is referred to as a data point, and the data points are numbered from earliest to latest according to the time when they were collected. For example, in the above example, the first collected current value of 13 is recorded as data point 1, which can be recorded as x1. The subsequently collected current value of 13.5 is recorded as data point 2, which can be recorded as x2, and so on. In addition, for each data point, the time when the data point is collected is recorded as the timestamp of the data point. The timestamp can be represented by t, and the timestamp number is the number of the corresponding data point. For example, the timestamp corresponding to x1 is 0:0:00 on July 1, recorded as t1; the timestamp corresponding to x2 is 0:0:01 on July 1, recorded as t2.

[0052] In combination with the above definition, the embodiment of the present application provides a method for cleaning time series industrial control protocol data, see Figure 1 , is a flowchart of the method, which may include the following steps.

[0053] The method provided in this embodiment can be executed by an industrial control system.

[0054] S101, obtaining time series data to be cleaned, historical time series data and maximum speed constraints.

[0055] The time series data to be cleaned may be original time series data collected by a sensor, for example, time series data composed of current values ​​at the aforementioned multiple moments.

[0056] In a specific application scenario, the industrial control system can control the sensor to collect the above-mentioned time series data to be cleaned based on a pre-configured industrial control protocol, and then clean the collected time series data to be cleaned through the cleaning method provided in this embodiment, and then use the cleaned time series data to perform subsequent analysis and processing processes.

[0057] Therefore, the time series data to be cleaned collected by the industrial control system in this embodiment can also be called time series industrial control protocol data.

[0058] The maximum speed constraint is a speed constraint set in advance based on experience and the source of the time series data to be cleaned. For example, if the time series data to be cleaned includes the electric power data consumed by a household in a community at various times of the day collected in real time, then it can be predicted based on experience that the fluctuation range of the household's electricity consumption during the day will not exceed a specific range, and therefore the maximum speed constraint is set based on this range.

[0059] Global speed constraints, including a global speed cap s max and the global speed lower limit s min .

[0060] The historical time series data is time series data that is collected before the time series data to be cleaned is collected and has the same source as the time series data to be cleaned.

[0061] For example, if the time series data to be cleaned includes the electric power consumption data of a household in a community collected in real time at various times on a certain day, then the historical time series data may include the electric power data of other households in the same community at various times on each of the previous days.

[0062] S102 : determining a current speed constraint based on data points of the first N cleaning windows in the time series data to be cleaned, a maximum speed constraint, and historical time series data.

[0063] Among them, the cleaning window is a pre-set time window, and the length of the cleaning window can be set according to actual needs. For example, the length w of the cleaning window can be set to 5 seconds, then every 5 seconds in the time series data to be cleaned is equivalent to a cleaning window, that is, the 1st to 5th seconds are the first cleaning window, the 6th to 10th seconds are the second cleaning window, and so on.

[0064] N is a pre-set positive integer, and its specific value can be set as needed, for example, N is set to be equal to 5.

[0065] Combined with the above example, when the length w of the cleaning window is set to 5 seconds and the number of cleaning windows N is set to 5, the data points of the first N cleaning windows of the time series data to be cleaned in S102 are each data point collected within the first 25 seconds (i.e., the product of the length and the number) of the time series data to be cleaned. The first 25 seconds here start from the moment when the first data point in the time series data to be cleaned is collected.

[0066] The specific execution process of step S102 is as follows:

[0067] First, calculate the velocity of each data point in the historical time series data.

[0068] In time series data, for any two data points x i and x k , data point x i To data point x k The rate of change v ik It can be calculated using the following formula (1):

[0069] v ik =(x k -x i )÷(t k -t i )-----(Formula 1).

[0070] where t k and t iThe data points x k is the data point x i The timestamp when the data point x k is the data point x i The next data point, for example, data point x i is x1, x k When x2, the above data point x i To data point x k The rate of change of can be regarded as the data point x i The speed is recorded as v i .

[0071] The above definition of data point velocity is applicable to both historical time series data and time series data to be cleaned.

[0072] Secondly, the mean value u and standard deviation sigma of the speed of each data point in the historical time series data are statistically calculated, and the initial speed constraint is determined based on the confidence interval width corresponding to the preset target confidence level, as well as the mean value u and standard deviation sigma.

[0073] The speed of each data point in the historical time series data usually satisfies the normal distribution X~N(u, sigma 2 ), in this normal distribution, the mean u can be regarded as the mathematical expectation of the random variable speed, sigma 2 It can be regarded as the variance of the random variable. In this normal distribution, different confidence intervals [uz*sigma, u+z*sigma] can be calculated based on different confidence interval widths z. Different confidence interval widths correspond to different confidence levels, which can be denoted as P. The corresponding relationship between confidence interval widths and confidence levels can be shown in Table 1 below.

[0074] Table 1

[0075]

[0076] Therefore, the confidence interval width corresponding to the preset target confidence level can be selected in Table 1, and then the selected confidence interval width is substituted into the above confidence interval expression. Combined with the mean and standard deviation of the speed in the aforementioned historical time series data, a specific confidence interval can be calculated. This confidence interval is the initial speed constraint that needs to be determined in S101.

[0077] The specific value of the target confidence can be set as needed without limitation.

[0078] For example, the target confidence level can be set to 0.9, and the confidence interval width z corresponding to 0.9 can be found in Table 1, which is approximately equal to 1.64. Substituting this confidence interval width into the above confidence interval expression, the initial velocity constraint V0 = [vmin0 =u-1.64sigma,v max0 =u+1.64sigma], where v min0 represents the lower limit of the initial velocity constraint, v max0 Indicates the upper limit of the initial velocity constraint.

[0079] Finally, according to the data points of the first N cleaning windows in the time series data to be cleaned and the maximum speed constraint, the initial speed constraint is adjusted to obtain the current speed constraint.

[0080] In this step, the speed of each data point in the first N cleaning windows of the time series data to be cleaned is first calculated, and then it is determined whether the speed of each data point in the first N cleaning windows of the time series data to be cleaned meets the initial speed constraint, that is, whether the speed of each data point is within the range defined by the upper and lower limits of the initial speed constraint. If the speed of each data point in the first N cleaning windows meets the initial speed constraint, the initial speed constraint is directly determined as the current speed constraint V = [v min , v max ], step S102 ends; if the speed of at least one data point in the first N cleaning windows does not meet the initial speed constraint, the iterative speed constraint V1=[v min1 , v max1 ], where v min1 represents the lower limit of the iteration speed constraint, v max1 Represents the upper limit of the iteration speed constraint.

[0081]

[0082] After obtaining the iteration speed constraint, it is determined whether the speed of each data point in the first N cleaning windows of the time series data to be cleaned meets the iteration speed constraint at this time. If the speed of each data point in the first N cleaning windows meets the iteration speed constraint at this time, the iteration speed constraint at this time is determined as the current speed constraint, and step S102 ends. If the speed of at least one data point in the first N cleaning windows does not meet the iteration speed constraint at this time, the v in the above formula (2) is changed. min0 and v max0 are replaced with the lower and upper limits of the iterative speed constraint at this time, and then the new v is calculated according to formula (2): min1 and v max1 , thereby obtaining a new iteration speed constraint.

[0083] After obtaining the new iteration speed constraint, repeat the above process of judging whether the speed of each data point satisfies the iteration speed constraint at this time, and updating it according to formula (2) if it does not meet the constraint, and so on, until the iteration speed constraint after a certain update can make the speed of each data point in the first N cleaning windows meet the constraint. Then, the iteration speed constraint that the speed of each data point in the first N cleaning windows meets can be determined as the current speed constraint V = [v min , v max ].

[0084] S103 , starting from the N+1th cleaning window of the time series data to be cleaned, identifying whether each data point in the time series data to be cleaned is an abnormal data point one by one according to the speed of each data point and the current speed constraint, and cleaning the identified abnormal data points.

[0085] Abnormal data points. For each data point in the time series data to be cleaned, if the speed of the data point exceeds the range defined by the current speed constraint, that is, the speed of the data point is not within [v min , v max ], the data point is considered to be an abnormal data point. Conversely, if the speed of the data point is within the interval defined by the current speed constraint, the data point can be considered to be a normal data point.

[0086] Abnormal data points can be divided into two categories: isolated abnormal points and continuous abnormal points.

[0087] If the previous data point and the next data point of an abnormal data point are both normal data points, then this abnormal data point is an isolated abnormal data point and is recorded as an isolated abnormal point.

[0088] If multiple (referring to two or more) consecutive data points are all abnormal data points, then each of the multiple consecutive abnormal data points belongs to the above-mentioned continuous abnormal points.

[0089] In step S103, the specific cleaning method varies according to the type of the identified abnormal data points.

[0090] When the identified abnormal data points are isolated abnormal points, the process of cleaning the identified abnormal data points includes:

[0091] A1, determine the first cleaning window starting from the timestamp of the isolated outlier point.

[0092] For example, the timestamp of the isolated abnormal point is the 5th second, and the preset cleaning window length w is 5 seconds. Then, in step A1, the time period from the 5th second to the 10th second can be determined as the first cleaning window of the isolated abnormal point.

[0093] A2, based on other data points whose timestamps are within the first cleaning window, calculates a finite candidate set of isolated outliers.

[0094] The other data points in step A2 refer to data points that are different from isolated outliers. k Indicates the isolated outlier identified, then step A2 is equivalent to the timestamp in (t k , t k +w], the calculated x k A finite set of candidates.

[0095] x k A finite candidate set X k , defined as x k The minimum candidate set X k-min 、x k The maximum candidate set X k-max and x k The union of itself. k The minimum candidate set X k-min It can be calculated according to the following formula (3):

[0096] X k-min ={x i +s min ×(t k -t i )|t k <t i ≤t k +w,1≤i≤n}--(Formula 3).

[0097] From formula (3), we can see that x k The minimum candidate set X k-min The calculation process is:

[0098] Determine that each timestamp in the time series data to be cleaned is within the first cleaning window, that is, within (t k , t k +w] data point x in the interval i , then for each x that satisfies the above conditions i , multiply the lower limit of the global velocity constraint by the data point x i The difference between the timestamp of the isolated outlier point and the timestamp of the isolated outlier point, and then the product of the obtained value and x i Add together to obtain the minimum candidate set X k-min An element in , all timestamps are located at (t k , t k +w] interval are calculated according to this process, and the set of elements obtained is x k The minimum candidate set X k-min.

[0099] x k The maximum candidate set X k-max It can be calculated according to the following formula (4):

[0100] X k-max ={x i +s max ×(t k -t i )|t k <t i ≤t k +w,1≤i≤n}--(Formula 4).

[0101] From formula (4), we can see that x k The maximum candidate set X k-max The calculation process is:

[0102] Determine that each timestamp in the time series data to be cleaned is within the first cleaning window, that is, within (t k , t k +w] data point x in the interval i , then for each x that satisfies the above conditions i , multiply the upper limit of the global velocity constraint by the data point x i The difference between the timestamp of the isolated outlier point and the timestamp of the isolated outlier point, and then the product of the obtained value and x i Add together to obtain the maximum candidate set X k-max An element in , all timestamps are located at (t k , t k +w] interval are calculated according to this process, and the set of elements obtained is x k The maximum candidate set X k-max .

[0103] A3, replaces the isolated outlier points with the median of a finite candidate set of isolated outlier points to complete the cleaning of isolated outlier points.

[0104] One implementation of step A3 is to arrange the elements in the limited candidate set in order from small to large or from large to small, and then take the data in the middle to replace the isolated abnormal point.

[0105] For example, the data point x in the time series data to be cleaned k Identified as an isolated outlier, before cleaning x k =16, after the above steps, the median of the limited candidate set of isolated outliers is determined to be 14, so in step A3, x k Set to 14, thus completing the x k That is, before cleaning, x kThe value of x is 16. k The value of is 14.

[0106] For ease of understanding, the following example illustrates the process of cleaning isolated outliers. Figure 2 , which is a schematic diagram of cleaning isolated abnormal points provided by this embodiment.

[0107] Figure 2 In the coordinate system, the vertical axis represents the value of each data point in the time series data to be cleaned, for example, the value of x1 is 13, and the horizontal axis represents the collection time of each data point, that is, the timestamp of the data point. For example, the timestamp t1 of x1 is equal to 1, which means that the data point x1 is collected at the 1st second.

[0108] exist Figure 2 In the example, data point x3 in the time series data to be cleaned is identified as an isolated outlier. The length w of the cleaning window is preset to 4, and the timestamp t3 of the data point x3 is equal to 3. Therefore, according to the aforementioned step A1, the first cleaning window is first determined to be 3~3+4, that is, the time period from the 3rd second to the 7th second.

[0109] Then, according to step A2, the maximum candidate set and the minimum candidate set of x3 are calculated using the data points whose timestamps are in the interval (3, 7), thereby determining the finite candidate set of x3.

[0110] like Figure 2 As shown in the figure, the data points with timestamps in the interval (3, 7] in the time series data to be cleaned include data points x4 to x6. Based on these data points, the maximum candidate set is calculated to be (20.5, 19, 13), and the minimum candidate set is calculated to be (13, 12.5, 9). The maximum candidate set and the minimum candidate set of x3 are combined with x3 to obtain the finite candidate set of x3: (20.5, 19, 14, 13, 13, 12.5, 9). Finally, the median of the finite candidate set is determined to be 13. Therefore, 13 is used to replace the original value 14 of x3, completing the cleaning of the isolated outlier.

[0111] When the identified abnormal data points are continuous abnormal points, the process of cleaning the identified abnormal data points includes:

[0112] B1, determine the abnormal sequence consisting of consecutive abnormal points.

[0113] An abnormal sequence is defined as a sequence consisting of multiple consecutive data points in the time series data to be cleaned and meeting the following conditions:

[0114] Every data point in the sequence is an abnormal data point, and the previous data point in the sequence is a normal data point, and the next data point in the sequence is also a normal data point.

[0115] For example, suppose x m is a normal data point, x m+1 to x m+p are all abnormal data points, x m+p+1 is a normal data point, then x m+1 to x m+p The composed sequence is an abnormal sequence.

[0116] That is to say, an abnormal sequence can be understood as a sequence that is completely composed of continuous abnormal points, and neither the previous data point nor the next data point of the sequence is an abnormal data point.

[0117] B2, calculate the change speed from the first reference point to the second reference point of the abnormal sequence.

[0118] The first reference point is the previous data point of the abnormal sequence, and the second reference point is the next data point of the abnormal sequence.

[0119] Combined with the example of step B1, for the abnormal sequence x m+1 to x m+p , its previous data point x m It is the first reference point, and the next data point x m+p+1 It is the second reference point.

[0120] The change speed from the first reference point to the second reference point can be calculated using the formula (1) used to calculate the change speed in the aforementioned step S102, and will not be repeated here.

[0121] B3, using the change speed from the first reference point to the second reference point to perform linear cleaning on each continuous abnormal point in the abnormal sequence.

[0122] The process of linear cleaning is as follows:

[0123] For each continuous abnormal point in the abnormal sequence, the time difference between the timestamp of the continuous abnormal point and the timestamp of the first reference point is calculated. The time difference is multiplied by the change speed from the first reference point to the second reference point to obtain the repair value of the continuous abnormal point. Finally, the repair value of the continuous abnormal point is used to replace the original value of the continuous abnormal point to complete the repair of the continuous abnormal point.

[0124] The following is an example to illustrate the process of repairing consecutive outliers. Figure 3 , is a schematic diagram of cleaning continuous abnormal points provided by the embodiment of the present application. Figure 3It can be seen that after cleaning all continuous abnormal points in the abnormal sequence according to the above process, the cleaned data points form a line segment with the first reference point and the second reference point as endpoints. The slope of the line segment is the change rate from the first reference point to the second reference point calculated previously. It can also be seen that the corresponding values ​​of the data points cleaned according to the above method are quite close to the true values ​​of these data points, that is, the actual values ​​of the parameters monitored at the same time, indicating that the above process of cleaning continuous abnormal points has high accuracy.

[0125] According to the above method for cleaning isolated abnormal points and continuous abnormal points, a specific implementation of step S103 may be:

[0126] Before starting to execute S103, a delayed repair queue is first established. During the execution of S103, if it is found that the currently identified data point is an abnormal data point and the previous data point of this data point is a normal data point, the abnormal data point and its previous data point are recorded in the delayed repair queue; if it is found that the currently identified data point and its previous data point are both abnormal data points, the abnormal data point is recorded in the delayed repair queue; if it is found that the currently identified data point is a normal data point and its previous data point is an abnormal data point, the recording is stopped to determine the length of the delayed repair queue at this time, that is, to determine the number of data points recorded in the delayed repair queue at this time.

[0127] If the length of the delayed repair queue is equal to 2 when recording is stopped, it means that there is only one abnormal data point in the delayed repair queue at this time, and the previous data point and the next data point of the abnormal data point are both normal data points. Therefore, the abnormal data point in the queue at this time is cleaned according to the aforementioned process of cleaning isolated abnormal points.

[0128] If the length of the delay repair queue is greater than 2 when recording is stopped, it means that there are multiple continuous abnormal points recorded in the delay repair queue at this time, so each continuous abnormal point in the queue at this time is cleaned according to the above-mentioned process of cleaning continuous abnormal points.

[0129] After each cleaning, the delayed repair queue is cleared. When abnormal data points are found later, the above process is repeated until the entire time series data to be cleaned is identified.

[0130] In some optional embodiments, in order to further improve the accuracy of the cleaning solution of this embodiment, the current speed constraint can be further dynamically adjusted during the implementation of the method. That is, after step S103, this embodiment can further include the following steps.

[0131] S104 , after each data point is identified, if the speed of the data point or the acceleration of multiple data points before the data point meets a preset update condition, update the current speed constraint according to the speed of the data point and the acceleration of multiple data points before the data point.

[0132] The specific execution process of step S104 includes:

[0133] First, determine whether the currently identified data point meets the update conditions.

[0134] The update conditions include the following two:

[0135] Condition 1: The direction of the acceleration of each data point in the adjustment window is the same, that is, the acceleration of each data point in the adjustment window is positive or negative.

[0136] Condition 2: The speed of the currently identified data point approaches the upper or lower limit of the current speed constraint.

[0137] In S104 , as long as the currently identified data point satisfies at least one of the first and second conditions, it is considered that the data point satisfies the update condition, and it is determined that the current speed constraint needs to be updated.

[0138] The adjustment window in condition 1 refers to the time window up to the timestamp of the currently identified data point, and the length is w. For example, the currently identified data point is x k , the corresponding timestamp is t k , then the adjustment window is [t k -w,t k ] This time interval. The data points within the adjustment window refer to the data points whose corresponding timestamps are within the adjustment window.

[0139] For any data point x in the time series data to be cleaned i , the acceleration A of the data point i It can be calculated using the following formula (5):

[0140]

[0141] Combined with the above definition of the speed of a data point, it can be considered that the acceleration of a data point is equal to the difference between the speed of the data point and the speed of the previous data point, divided by the difference between the timestamp of the data point and the timestamp of the previous data point.

[0142] Approximation in condition 2 can be defined as the distance between the speed of the currently identified data point and the upper or lower limit of the current speed constraint being less than a preset update threshold. For example, the update threshold can be set to 0.2, or another value. If the distance between the speed of the currently identified data point and the upper limit of the current speed constraint is less than 0.2, the speed of the currently identified data point is considered to be approaching the upper limit of the current speed constraint.

[0143] For the sake of distinction, the current speed constraint is recorded as V(k) = [v(k) min , v(k) max ], where k represents the current speed constraint after the kth update, that is, the current speed constraint obtained after the kth execution of step S104, v(k) min and v(k) max are the upper and lower limits of the current speed constraint after the kth update. For example, the current speed constraint obtained after the first execution of S104 can be recorded as V(1)=[v(1) min ,v(1) max ].

[0144] In particular, the current speed constraint obtained for the first time after executing step S102 can be recorded as V(0)=[v(0) min ,v(0) max ], indicating the current speed constraint which has not been updated according to S104.

[0145] The process of updating the current velocity constraint is as follows:

[0146] Calculate the acceleration of each data point in the adjustment window, and calculate the velocity constraint update Delta-v based on the acceleration and weight of these data points.

[0147] Specifically, the speed constraint update amount can be calculated using the following formula (6):

[0148]

[0149] Formula (6) indicates that for each data point in the adjustment window except the currently identified data point, that is, for each time stamp between [t k -w,t k )Data point x in the interval p , set a weight B for the data point p Then use the weight B of the data point p , acceleration A p and timestamp t p Multiply to get the product of the data point, and finally get all the timestamps between [t k -w,t k) The product of the data points in the interval is added, and the result is the speed constraint update Delta-v used in this update.

[0150] The weight configuration of each data point in the adjustment window can be set according to the actual situation. Generally, it can be determined based on the distance of each data point relative to the currently identified data point. p The timestamp distance from the currently identified data point x k The farther the timestamp is, the more weight B p The smaller the data point x p The timestamp distance from the currently identified data point x k The closer the timestamp is, the greater the corresponding weight B p The bigger it is.

[0151] Finally, the current speed constraint is updated using the speed constraint update amount to obtain an updated current speed constraint.

[0152] Specifically, the current speed constraint can be updated according to the following formula (7):

[0153]

[0154] In formula (7), v(k) min and v(k) max They are the upper and lower limits of the current speed constraint after this update, i.e. after the kth update. v(k-1) min and v(k-1) max are the upper and lower limits of the current speed constraint after the last update, i.e. before this (kth) update. For example, if S104 is executed for the second time, the upper and lower limits of the current speed constraint after this update are v(2). min and v(2) max , the upper and lower limits of the current speed constraint before this update are v(1) min and v(1) max .

[0155] The present application provides a method for cleaning time-series industrial control protocol data, the method comprising: obtaining time-series data to be cleaned, historical time-series data from the same data source, and a maximum speed constraint; determining a current speed constraint based on the data points, maximum speed constraint, and historical time-series data in the first N cleaning windows of the time-series data to be cleaned; starting from the N+1th cleaning window of the time-series data to be cleaned, identifying whether each data point in the time-series data to be cleaned is an abnormal data point one by one based on the speed of each data point and the current speed constraint, and cleaning the identified abnormal data points. This solution determines a matching current speed constraint based on the time-series data to be cleaned, and identifies and cleans abnormal data points based on whether the speed of the data point meets the current speed constraint. Even if the time-series data has large fluctuations, it can accurately and clean the abnormal data points therein, thereby improving the accuracy of data cleaning.

[0156] Furthermore, compared with the smoothing-based cleaning algorithm, the present invention can more accurately identify abnormal error points, will not over-clean the original data points, and the values ​​of the root mean square error and Euclidean distance are reduced by at least 3 times compared with the smoothing algorithm, thereby having a better cleaning effect.

[0157] Compared with the data cleaning algorithm SCREEN based on speed constraints, the speed constraints of the present invention can be dynamically adjusted according to the speed change trend, can effectively handle abnormal small error points and continuous abnormal error points, and have better cleaning accuracy.

[0158] According to the method for cleaning time-series industrial control protocol data provided by the embodiment of the present application, the embodiment of the present application also provides a device for cleaning time-series industrial control protocol data, see Figure 4 , is a structural diagram of the device, which may include the following units.

[0159] An obtaining unit 401 is configured to obtain time series data to be cleaned, historical time series data, and a maximum speed constraint; wherein the historical time series data and the time series data to be cleaned have the same data source;

[0160] A determination unit 402 is configured to determine a current speed constraint based on data points in the first N cleaning windows of the time series data to be cleaned, the maximum speed constraint, and the historical time series data; wherein N is a preset positive integer, and the historical time series data and the time series data to be cleaned have the same data source;

[0161] The cleaning unit 403 is used to identify whether each data point in the time series data to be cleaned is an abnormal data point one by one according to the speed of each data point and the current speed constraint, starting from the N+1th cleaning window of the time series data to be cleaned, and clean the identified abnormal data points.

[0162] Optionally, when the determining unit 402 determines the current speed constraint based on the data points of the first N cleaning windows in the time series data to be cleaned, the maximum speed constraint, and the historical time series data, it is specifically configured to:

[0163] Calculate the velocity of each data point in the historical time series data;

[0164] Calculate the mean and standard deviation of the speed of each data point in the historical time series data, and determine the initial speed constraint based on the confidence interval width corresponding to the preset target confidence level, as well as the mean and standard deviation;

[0165] The initial speed constraint is adjusted according to the data points of the first N cleaning windows in the time series data to be cleaned and the maximum speed constraint to obtain the current speed constraint.

[0166] Optionally, when the identified abnormal data point is an isolated abnormal point, the cleaning unit 403 cleans the identified abnormal data point by:

[0167] Determine a first cleaning window starting from the timestamp of the isolated outlier point;

[0168] A finite candidate set of isolated outlier points is calculated based on other data points whose timestamps are within the first cleaning window; wherein the other data points refer to data points different from the isolated outlier points;

[0169] The isolated outlier points are replaced by the median of a finite candidate set of isolated outlier points to complete the cleaning of isolated outlier points.

[0170] Optionally, when the identified abnormal data points are continuous abnormal points, the cleaning unit 403 cleans the identified abnormal data points by:

[0171] Determine an abnormal sequence consisting of consecutive abnormal points; wherein an abnormal sequence refers to a sequence consisting entirely of consecutive abnormal points, and neither the previous data point nor the next data point of the sequence is an abnormal data point;

[0172] Calculate the change rate from the first reference point to the second reference point of the abnormal sequence; wherein the reference point is the previous data point of the abnormal sequence, and the second reference point is the next data point of the abnormal sequence;

[0173] Each continuous abnormal point in the abnormal sequence is linearly cleaned using the change speed from the first reference point to the second reference point.

[0174] Optionally, the apparatus further includes an updating unit 404, configured to:

[0175] After each data point is identified, if the speed of the data point or the acceleration of multiple data points before the data point meets the preset update conditions, the current speed constraint is updated according to the speed of the data point and the acceleration of multiple data points before the data point.

[0176] The specific working principle and beneficial effects of the cleaning device for time-series industrial control protocol data provided in this embodiment can be found in the relevant steps and beneficial effects of the cleaning method for time-series industrial control protocol data provided in the embodiment of this application, and will not be repeated here.

[0177] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0178] It should be noted that the concepts of "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0179] The present application is capable of being implemented or used by those skilled in the art. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to be embodied in the widest possible manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for cleaning sequential industrial control protocol data, characterized in that: include: Obtaining time series data to be cleaned, historical time series data, and a maximum speed constraint; wherein the historical time series data and the time series data to be cleaned have the same data source; Determine a current speed constraint based on data points of the first N cleaning windows in the time series data to be cleaned, the maximum speed constraint, and the historical time series data; wherein N is a preset positive integer; Starting from the N+1th cleaning window of the time series data to be cleaned, identifying whether each data point in the time series data to be cleaned is an abnormal data point one by one according to the speed of each data point and the current speed constraint, and cleaning the identified abnormal data points; After each data point is identified, if the speed of the data point or the acceleration of multiple data points before the data point meets a preset update condition, the current speed constraint is updated according to the speed of the data point and the acceleration of multiple data points before the data point; The determining of the current speed constraint based on the data points of the first N cleaning windows in the time series data to be cleaned, the maximum speed constraint, and the historical time series data includes: Calculating the velocity of each data point in the historical time series data; Counting the average value and standard deviation of the speed of each data point in the historical time series data, and determining the initial speed constraint according to the confidence interval width corresponding to the preset target confidence level, as well as the average value and the standard deviation; Adjust the initial speed constraint according to the data points of the first N cleaning windows in the time series data to be cleaned and the maximum speed constraint to obtain a current speed constraint; The update conditions include: Condition 1: The direction of the acceleration of each data point in the adjustment window is the same, and the adjustment window refers to a time window with a preset length ending at the timestamp of the currently identified data point; Condition 2: The distance between the speed of the currently identified data point and the upper limit or lower limit of the current speed constraint is less than a preset update threshold.

2. The method according to claim 1, characterized in that When the identified abnormal data point is an isolated abnormal point, cleaning the identified abnormal data point includes: determining a first cleaning window starting from the timestamp of the isolated outlier point; Calculating a finite candidate set of the isolated outlier points based on other data points whose timestamps are within the first cleaning window; wherein the other data points refer to data points different from the isolated outlier points; The isolated outlier point is replaced by the median of the finite candidate set of the isolated outlier point to complete the cleaning of the isolated outlier point.

3. The method according to claim 1, characterized in that When the identified abnormal data points are continuous abnormal points, cleaning the identified abnormal data points includes: Determine an abnormal sequence consisting of continuous abnormal points; wherein the abnormal sequence refers to a sequence consisting entirely of continuous abnormal points, and neither the previous data point nor the next data point of the sequence is an abnormal data point; Calculating a change rate from a first reference point to a second reference point of the abnormal sequence; wherein the first reference point is a previous data point of the abnormal sequence, and the second reference point is a subsequent data point of the abnormal sequence; Each continuous abnormal point in the abnormal sequence is linearly cleaned using the change speed from the first reference point to the second reference point.

4. A cleaning device for sequential industrial control protocol data, characterized in that: include: an obtaining unit, configured to obtain time series data to be cleaned, historical time series data, and a maximum speed constraint; wherein the historical time series data and the time series data to be cleaned have the same data source; a determining unit, configured to determine a current speed constraint based on data points of the first N cleaning windows in the time series data to be cleaned, the maximum speed constraint, and historical time series data; wherein N is a preset positive integer, and the historical time series data and the time series data to be cleaned have the same data source; a cleaning unit, configured to, starting from the N+1th cleaning window of the time series data to be cleaned, identify whether each data point in the time series data to be cleaned is an abnormal data point one by one according to the speed of each data point and the current speed constraint, and clean the identified abnormal data points; an updating unit, configured to update the current speed constraint according to the speed of the data point and the acceleration of the data points before the data point after each data point is identified, if the speed of the data point or the acceleration of the data points before the data point meets a preset update condition; The determining unit determines the current speed constraint based on the data points of the first N cleaning windows in the time series data to be cleaned, the maximum speed constraint, and the historical time series data, specifically for: Calculating the velocity of each data point in the historical time series data; Counting the average value and standard deviation of the speed of each data point in the historical time series data, and determining the initial speed constraint according to the confidence interval width corresponding to the preset target confidence level, as well as the average value and the standard deviation; Adjust the initial speed constraint according to the data points of the first N cleaning windows in the time series data to be cleaned and the maximum speed constraint to obtain a current speed constraint; The update conditions include: Condition 1: The direction of the acceleration of each data point in the adjustment window is the same, and the adjustment window refers to a time window with a preset length ending at the timestamp of the currently identified data point; Condition 2: The distance between the speed of the currently identified data point and the upper limit or lower limit of the current speed constraint is less than a preset update threshold.

5. The device according to claim 4, characterized in that When the identified abnormal data point is an isolated abnormal point, the cleaning unit is specifically used to clean the identified abnormal data point: determining a first cleaning window starting from the timestamp of the isolated outlier point; Calculating a finite candidate set of the isolated outlier points based on other data points whose timestamps are within the first cleaning window; wherein the other data points refer to data points different from the isolated outlier points; The isolated outlier point is replaced by the median of the finite candidate set of the isolated outlier point to complete the cleaning of the isolated outlier point.

6. The device according to claim 4, characterized in that When the identified abnormal data points are continuous abnormal points, the cleaning unit is specifically used to clean the identified abnormal data points: Determine an abnormal sequence consisting of continuous abnormal points; wherein the abnormal sequence refers to a sequence consisting entirely of continuous abnormal points, and neither the previous data point nor the next data point of the sequence is an abnormal data point; Calculating a change rate from a first reference point to a second reference point of the abnormal sequence; wherein the first reference point is a previous data point of the abnormal sequence, and the second reference point is a subsequent data point of the abnormal sequence; Each continuous abnormal point in the abnormal sequence is linearly cleaned using the change speed from the first reference point to the second reference point.

Images