Data cleaning method and device based on time sequence similarity, medium and equipment

Abstract

The invention provides a data cleaning method and device based on time sequence similarity, a medium and equipment. The method comprises the steps that firstly, a to-be-cleaned time sequence data set and multiple historical time sequence curves are acquired; discretizing the to-be-cleaned time sequence data based on a symbol clustering method to obtain a character sequence data set; performing similarity measurement on the character sequence data set and a historical character sequence data set corresponding to the historical time sequence curve by using an Euclidean distance algorithm, and determining a target time sequence curve; and finally, comparing the target time sequence curve with to-be-cleaned time sequence data in the to-be-cleaned time sequence data set, and determining abnormal data points. And correcting the time sequence data corresponding to the abnormal data points based on the target time sequence curve to obtain a target time sequence data set. According to the invention, through dimension reduction, similarity measurement optimization and accurate identification of abnormal data, missing data is reasonably filled up, and the accuracy and efficiency of power load data cleaning are improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data cleaning, and in particular to a data cleaning method, device, medium and equipment based on time series similarity. Background Art

[0002] In the field of information technology and data processing, with the rapid development of smart grid, Internet of Things and big data technology, the processing and analysis of power load data has become particularly critical. Power load data reflects the operating status of the power system and the power consumption behavior of users. Therefore, power load data plays an important role in the operation optimization, energy efficiency management and fault prediction of the power system. However, due to the characteristics of massive, high-dimensional and strong real-time nature of power load data, its processing and analysis face many challenges. Summary of the invention

[0003] The embodiments of the present disclosure at least provide a data cleaning method, device, medium and equipment based on time series similarity, which improves the accuracy and efficiency of power load data cleaning by reducing data dimensions, improving the efficiency of similarity measurement, and realizing accurate identification of abnormal data and reasonable filling of missing data.

[0004] The present disclosure provides a data cleaning method based on time series similarity, including:

[0005] Acquire a time series data set to be cleaned and a plurality of historical time series curves, wherein the time series data set to be cleaned includes time series data to be cleaned at different times in the power system; and discretize the time series data to be cleaned based on a symbol clustering method to obtain a character sequence data set, wherein the character sequence data set includes a plurality of character sequence data;

[0006] Using the Euclidean distance algorithm to measure the similarity between the character sequence data set and the historical character sequence data set corresponding to the historical timing curve, and determine the target timing curve;

[0007] The target time series curve is compared with the time series data to be cleaned in the time series data set to be cleaned to determine abnormal data points; and the time series data corresponding to the abnormal data points are corrected based on the target time series curve to obtain the target time series data set.

[0008] In some possible embodiments, before obtaining the time series data set to be cleaned and the plurality of historical time series curves, the process includes:

[0009] Acquire a power load statistical data set to be cleaned; the power load statistical data set includes a power load time series data set corresponding to a plurality of preset variables;

[0010] Based on the power load statistical data set, the maximum information coefficient between each preset variable and the power load is calculated as the correlation between each preset variable and the power load, the target variable among the preset variables is determined based on the correlation, and the power load time series data set corresponding to the target variable is obtained as the time series data set to be cleaned.

[0011] In some possible embodiments, the discretization processing of the time series data to be cleaned based on the symbolic clustering method includes:

[0012] The time series data to be cleaned are respectively normalized according to the data normalization formula, and the normalized data are respectively reduced in dimension based on the dimension reduction formula to obtain a reduced-dimension sequence data set; in the reduced-dimension sequence data set, the normalized data are distributed into a plurality of data intervals;

[0013] Performing symbol clustering processing on the reduced dimension sequence data set based on a preset symbol clustering standard to obtain the character sequence data set; the preset symbol clustering standard includes representing data in the same data interval in the reduced dimension sequence data set with the same character;

[0014] The data normalization formula includes:

[0015]

[0016] Among them, c i represents the i-th data in the dimensionality reduction sequence data set; x i represents the i-th data in the time series data set to be cleaned; μ(X) represents the average value of all the time series data to be cleaned in the time series data set to be cleaned; δ represents the standard deviation of all the time series data to be cleaned in the time series data set to be cleaned;

[0017] The dimension reduction processing formula includes:

[0018]

[0019] in, represents the i-th data of the character sequence data set; n represents the data length of the time series data set to be cleaned; N represents the number of segments represented by the dimension-reduced sequence data set; c j Represents the i-th data of the character sequence data set.

[0020] In some possible embodiments, before using the Euclidean distance algorithm to measure the similarity between the character sequence data set and the historical character sequence data set corresponding to the historical time series curve, the method includes:

[0021] A historical sequence data set corresponding to each historical time series curve is obtained, and the historical sequence data sets are discretized based on the symbol clustering method to obtain the historical character sequence data set.

[0022] In some possible embodiments, the using of the Euclidean distance algorithm to measure the similarity between the character sequence data set and the historical character sequence data set corresponding to the historical timing curve to determine the target timing curve includes:

[0023] Determine the Euclidean values ​​between the character sequence data set and each of the historical character sequence data sets based on the Euclidean distance calculation formula; and determine the similarity between the character sequence data set and each of the historical character sequence data sets according to the calculation result;

[0024] Determining a plurality of similar time series curves corresponding to the character sequence data set based on the similarity between the character sequence data set and each of the historical character sequence data sets;

[0025] Performing curve weighting processing on the plurality of similar timing curves according to a preset curve adjustment method to obtain a target timing curve;

[0026] The Euclidean distance calculation formula includes:

[0027]

[0028] Among them, L q,p represents the Euclidean value between the character sequence dataset q and the historical character sequence dataset p; q i represents the i-th data in the character sequence data set q; p i Represents the i-th data in the historical character sequence dataset p.

[0029] In some possible embodiments, comparing the target time series curve with the time series data to be cleaned in the time series data set to be cleaned to determine abnormal data points includes:

[0030] Calculate the average value of the data in a fixed time period corresponding to the target time series curve, and use the calculation result as the judgment value;

[0031] Determine an abnormal data judgment rule according to the judgment value, and respectively judge whether each time series data to be cleaned in the time series data set to be cleaned meets the abnormal data judgment rule, if not, the time series data to be cleaned is an abnormal data point;

[0032] The abnormal data judgment rules include:

[0033]

[0034] in, Indicates the judgment value; x k Represents the kth time series data to be cleaned in the time series data set to be cleaned; Indicates the judgment threshold of the kth data; a k1 ,a k2 ,...,a kw represents the kth data in w historical character sequence data sets. In some possible embodiments,

[0035] In some possible embodiments, before respectively determining whether each time series data to be cleaned in the time series data set to be cleaned satisfies the abnormal data judgment rule, the method further includes:

[0036] Determine whether there are missing values ​​in each time series data to be cleaned in the time series data set to be cleaned, and if so, complete the time series data at the location of the missing value in the time series data set to be cleaned based on the determination value;

[0037] Accordingly, the method of correcting the time series data corresponding to the abnormal data point based on the target time series curve includes:

[0038] The time series data at the location where the abnormal data point is located in the time series data set to be cleaned is corrected based on the judgment value.

[0039] The present disclosure provides a data cleaning device based on time series similarity, including:

[0040] A discretization processing module is used to obtain a time series data set to be cleaned and a plurality of historical time series curves, wherein the time series data set to be cleaned includes time series data to be cleaned at different times in the power system; and discretize the time series data to be cleaned based on a symbol clustering method to obtain a character sequence data set, wherein the character sequence data set includes a plurality of character sequence data;

[0041] A similarity measurement module, used to measure the similarity between the character sequence data set and the historical character sequence data set corresponding to the historical timing curve using the Euclidean distance algorithm to determine the target timing curve;

[0042] The data correction module is used to compare the target time series curve with the time series data to be cleaned in the time series data set to be cleaned to determine abnormal data points; and based on the target time series curve, the time series data corresponding to the abnormal data points are corrected to obtain the target time series data set.

[0043] In some possible embodiments, the discretization processing module is further used for:

[0044] Acquire a power load statistical data set to be cleaned; the power load statistical data set includes a power load time series data set corresponding to a plurality of preset variables;

[0045] Based on the power load statistical data set, the maximum information coefficient between each preset variable and the power load is calculated as the correlation between each preset variable and the power load, the target variable among the preset variables is determined based on the correlation, and the power load time series data set corresponding to the target variable is obtained as the time series data set to be cleaned.

[0046] In some possible embodiments, the discretization processing module is specifically used to:

[0047] The time series data to be cleaned are respectively normalized according to the data normalization formula, and the normalized data are respectively reduced in dimension based on the dimension reduction formula to obtain a reduced-dimension sequence data set; in the reduced-dimension sequence data set, the normalized data are distributed into a plurality of data intervals;

[0048] Performing symbol clustering processing on the reduced dimension sequence data set based on a preset symbol clustering standard to obtain the character sequence data set; the preset symbol clustering standard includes representing data in the same data interval in the reduced dimension sequence data set with the same character;

[0049] The data normalization formula includes:

[0050]

[0051] Among them, c i represents the i-th data in the dimensionality reduction sequence data set; x i represents the i-th data in the time series data set to be cleaned; μ(X) represents the average value of all the time series data to be cleaned in the time series data set to be cleaned; δ represents the standard deviation of all the time series data to be cleaned in the time series data set to be cleaned;

[0052] The dimension reduction processing formula includes:

[0053]

[0054] in, represents the i-th data of the character sequence data set; n represents the data length of the time series data set to be cleaned; N represents the number of segments represented by the dimension-reduced sequence data set; c j Represents the i-th data of the character sequence data set.

[0055] In some possible embodiments, the similarity measurement module is further used for:

[0056] A historical sequence data set corresponding to each historical time series curve is obtained, and the historical sequence data sets are discretized based on the symbol clustering method to obtain the historical character sequence data set.

[0057] In some possible embodiments, the similarity measurement module is specifically used to:

[0058] Determine the Euclidean values ​​between the character sequence data set and each of the historical character sequence data sets based on the Euclidean distance calculation formula; and determine the similarity between the character sequence data set and each of the historical character sequence data sets according to the calculation result;

[0059] Determining a plurality of similar time series curves corresponding to the character sequence data set based on the similarity between the character sequence data set and each of the historical character sequence data sets;

[0060] Performing curve weighting processing on the plurality of similar timing curves according to a preset curve adjustment method to obtain a target timing curve;

[0061] The Euclidean distance calculation formula includes:

[0062]

[0063] Among them, L q,p represents the Euclidean value between the character sequence dataset q and the historical character sequence dataset p; q i represents the i-th data in the character sequence data set q; p i Represents the i-th data in the historical character sequence dataset p.

[0064] In some possible embodiments, the data correction module is specifically used to:

[0065] Calculate the average value of the data in a fixed time period corresponding to the target time series curve, and use the calculation result as the judgment value;

[0066] Determine an abnormal data judgment rule according to the judgment value, and respectively judge whether each time series data to be cleaned in the time series data set to be cleaned meets the abnormal data judgment rule, if not, the time series data to be cleaned is an abnormal data point;

[0067] The abnormal data judgment rules include:

[0068]

[0069] Among them, x represents the judgment value; x k Represents the kth time series data to be cleaned in the time series data set to be cleaned; Indicates the judgment threshold of the kth data; a k1 ,a k2 ,...,a kw Represents the kth data in the w historical character sequence data set.

[0070] In some possible embodiments, the data correction module is further used to:

[0071] Determine whether there are missing values ​​in each time series data to be cleaned in the time series data set to be cleaned, and if so, complete the time series data at the location of the missing value in the time series data set to be cleaned based on the determination value;

[0072] Accordingly, the data correction module is specifically used for:

[0073] The time series data at the location where the abnormal data point is located in the time series data set to be cleaned is corrected based on the judgment value.

[0074] An embodiment of the present disclosure provides a computer device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the data cleaning method based on time series similarity as described in any possible implementation manner described above is performed.

[0075] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the data cleaning method based on time series similarity as described in any possible implementation manner described above is implemented.

[0076] The data cleaning method, device, medium and equipment based on time series similarity provided in the embodiments of the present disclosure. First, a time series data set to be cleaned and multiple historical time series curves are obtained; and the time series data to be cleaned are discretized based on the symbol clustering method to obtain a character sequence data set; then the Euclidean distance algorithm is used to measure the similarity between the character sequence data set and the historical character sequence data set corresponding to the historical time series curve to determine the target time series curve; finally, the target time series curve is compared with the time series data to be cleaned in the time series data set to be cleaned to determine the abnormal data points; and the time series data corresponding to the abnormal data points are corrected based on the target time series curve to obtain the target time series data set. In this way, the accuracy and efficiency of power load data cleaning are improved by reducing the data dimension, improving the efficiency of similarity measurement, and realizing the accurate identification of abnormal data and the reasonable filling of missing data.

[0077] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required to be cited in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0079] Figure 1 A flow chart of a data cleaning method based on time series similarity provided by an embodiment of the present disclosure is shown;

[0080] Figure 2 A flow chart of a data discretization processing method in a data cleaning method based on time series similarity provided by an embodiment of the present disclosure is shown;

[0081] Figure 3 A schematic diagram of a data cleaning method based on time series similarity provided by an embodiment of the present disclosure is shown;

[0082] Figure 4 A schematic diagram of the structure of a data cleaning device based on time series similarity provided by an embodiment of the present disclosure is shown;

[0083] Figure 5 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0084] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0085] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0086] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0087] In the field of information technology and data processing, with the rapid development of smart grid, Internet of Things and big data technology, the processing and analysis of power load data has become an important research direction. Power load data reflects the key information of power system operation status and user power consumption behavior, which is of great significance for power system operation optimization, energy efficiency management, fault prediction, etc. However, due to its massive, high-dimensional and strong real-time characteristics, the processing and analysis of power load data faces many challenges.

[0088] Research has found that the processing of power load data mainly relies on traditional data processing technologies and methods, including data cleaning, data compression and dimensionality reduction, time series analysis, data mining and machine learning, and other key areas. Data cleaning improves the quality and availability of data by removing noise, processing missing values ​​and outliers, etc.; data compression and dimensionality reduction aim to reduce the cost and complexity of data processing; time series analysis technology is applied to load forecasting and characteristic analysis; and data mining and machine learning methods can perform deep mining and intelligent analysis. However, these methods generally have problems such as high computational complexity, low accuracy, and lack of adaptability when processing large-scale, high-dimensional power load data.

[0089] Based on the above research, a data cleaning method, device, medium and equipment based on time series similarity are provided in the embodiments of the present disclosure. First, a time series data set to be cleaned and multiple historical time series curves are obtained; and the time series data to be cleaned are discretized based on the symbol clustering method to obtain a character sequence data set; then the Euclidean distance algorithm is used to measure the similarity between the character sequence data set and the historical character sequence data set corresponding to the historical time series curve to determine the target time series curve; finally, the target time series curve is compared with the time series data to be cleaned in the time series data set to be cleaned to determine the abnormal data points; and the time series data corresponding to the abnormal data points are corrected based on the target time series curve to obtain the target time series data set.

[0090] In the disclosed embodiment, the power load data to be cleaned is discretized and reduced in dimension by using the symbol clustering method, and the data after the dimension reduction process is used to measure the similarity with the historical character sequence data set, and then the measurement result is compared with the original power load data set to determine the abnormal data points and correct them. In this way, the accuracy and efficiency of power load data cleaning are improved.

[0091] To facilitate understanding of this embodiment, the execution subject of the data cleaning method based on time series similarity provided by the embodiment of the present disclosure is first introduced in detail. The execution subject of the data cleaning method based on time series similarity provided by the embodiment of the present disclosure is a computer device. The computer device can be a server. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data and artificial intelligence platforms.

[0092] The data cleaning method based on time series similarity provided by the embodiment of the present application is described in detail below in conjunction with the accompanying drawings. Figure 1 As shown, it is a flow chart of a data cleaning method based on time series similarity provided by an embodiment of the present disclosure, and the method includes the following S101 to S103:

[0093] S101, obtaining a time series data set to be cleaned and multiple historical time series curves, wherein the time series data set to be cleaned includes time series data to be cleaned at different times in the power system; and discretizing the time series data to be cleaned based on a symbol clustering method to obtain a character sequence data set, wherein the character sequence data set includes multiple character sequence data.

[0094] It is understandable that in the power load statistics, since the final power load value is affected by many factors, such as temperature, climate, work and rest time, and electricity price, if all data are cleaned, the model structure will become complicated, which will also bring great difficulties to the future analysis of user energy consumption characteristics and hinder the improvement of data accuracy. Therefore, before obtaining the time series data set to be cleaned, the data in the power system can be preprocessed, including the following (1) to (2):

[0095] (1) obtaining a power load statistical data set to be cleaned; the power load statistical data set includes a power load time series data set corresponding to a plurality of preset variables;

[0096] (2) Based on the power load statistical data set, the maximum information coefficient between each preset variable and the power load is calculated as the correlation between each preset variable and the power load, the target variable among the preset variables is determined based on the correlation, and the power load time series data set corresponding to the target variable is obtained as the time series data set to be cleaned.

[0097] Here, the power load statistical data set obtained by the present invention contains power load data at 96 time points, and the length of the 96-point daily load data is fixed (the sampling period is 15 minutes, and the load sequence for each day is 96 points). The preset variables can be factors that may affect the power load results, such as temperature, climate, work and rest time or electricity price. The maximum information coefficient (MIC), also known as the maximum mutual information coefficient, is an important reference indicator for measuring the strength of the linear or nonlinear relationship between two variables. The value range of MIC is [0,1]. The higher the MIC value, the closer the relationship between the two variables. When MIC=0, it indicates that the two variables are independent of each other. After obtaining the power load time series data set corresponding to multiple preset variables, for each preset variable, it is necessary to calculate the maximum information coefficient between it and the power load as the correlation to screen out the factors that mainly act on the power load. Among them, the calculation formula of MIC is expressed as:

[0098]

[0099] Where D(X,Y) represents a set of n ordered pairs of preset variables X and power load Y; B(n) is a function of n, which is used to control the complexity of the grid and avoid overfitting. Usually, B(n) = n 0.6 ; I(D,X,Y) represents the mutual information between the preset variable X and the power load Y.

[0100] Specifically, after obtaining the correlation between each preset variable and the power load, one or several preset variables with a strong correlation with the power load can be determined as the target variable according to the correlation result, and the power load time series data set corresponding to the target variable is used as the time series data set to be cleaned to complete the subsequent data cleaning work. Here, since the MIC method is based on the characteristics of the data itself, through the correlation between the influencing factors (i.e., each preset variable) and the power load, the main influencing factors with a strong correlation with the power load can be adaptively screened, which improves the adaptability of the data cleaning method proposed in the present disclosure and simplifies the subsequent processing work.

[0101] It is understandable that after obtaining the time series data set to be cleaned, it is first discretized using the symbolic clustering method to achieve data dimensionality reduction. Among them, the symbolic clustering method (Symbolic Aggregate ApproXimation, SAX) is a technology that combines the ideas of time series data dimensionality reduction and cluster analysis for time series data analysis. Usually, time series data is composed of continuous discrete numerical points, and the SAX method can replace these numerical points with discrete sequences represented by characters. Its main purpose is to simplify the representation of the original time series data through this symbolic sequence to reduce the data dimension and reduce the impact of noise. Specifically, refer to Figure 2 As shown, when discretizing the time series data to be cleaned based on the symbolic clustering method, the following S1011 to S1012 may be included:

[0102] S1011, normalize the time series data to be cleaned according to the data normalization formula, and perform dimensionality reduction processing on the normalized data based on the dimensionality reduction processing formula to obtain a reduced dimensionality sequence data set; in the reduced dimensionality sequence data set, the normalized data is distributed into multiple data intervals.

[0103] Here, data normalization is to scale the data to a specific minimum and maximum value. The formula is:

[0104]

[0105] Among them, c i represents the i-th data in the dimensionality reduction sequence data set; x i represents the i-th data in the time series data set to be cleaned; μ(X) represents the average value of all the time series data to be cleaned in the time series data set to be cleaned; δ represents the standard deviation of all the time series data to be cleaned in the time series data set to be cleaned.

[0106] It is understandable that after the time series data to be cleaned is normalized, dimensionality reduction technology can be used to reduce the dimension of the data while retaining as much information as possible. Here, the present disclosure uses a piecewise aggregate approximation algorithm (PAA), the principle of which is to segment and reduce the dimension of the time series load data of length n to generate an N-dimensional time series data, and then separate m equal probability regions. If the values ​​in the sequence after dimensionality reduction correspond to the same probability interval, they are represented by the same symbol, so that the sequence is represented by discrete characters, achieving the purpose of reducing the dimension of the data after normalization and reducing noise. Therefore, the PAA algorithm uses a dimensionality reduction processing formula to reduce the dimension of the normalized data to obtain a reduced dimensionality sequence data set; its dimensionality reduction processing formula is expressed as:

[0107]

[0108] in, represents the i-th data of the character sequence data set; n represents the data length of the time series data set to be cleaned; N represents the number of segments represented by the dimension-reduced sequence data set; c j Represents the i-th data of the character sequence data set.

[0109] In some other embodiments, the method for performing data dimensionality reduction on a data set may also utilize methods such as principal component analysis or t-SNE, which are not specifically limited here.

[0110] S1012, performing symbol clustering processing on the reduced dimension sequence data set based on a preset symbol clustering standard to obtain the character sequence data set; the preset symbol clustering standard includes representing data in the same data interval in the reduced dimension sequence data set with the same character.

[0111] Specifically, the preset symbol clustering standard in the present disclosure includes representing the data of the same data interval in the reduced dimensionality sequence data set with the same character. Therefore, after obtaining the reduced dimensionality sequence data set, according to the selected size of the letter set, the interval splitting point β is found using the Gaussian distribution table. i ,Will Mapped to m intervals of equal probability, in the same interval Use the same character representation to obtain a character sequence data set

[0112] S102, using a Euclidean distance algorithm to measure the similarity between the character sequence data set and the historical character sequence data set corresponding to the historical time series curve, and determining a target time series curve.

[0113] It is understandable that the Euclidean distance is a similarity measurement tool. Its application requires that the compared sequences must have a corresponding length and the points where the difference between the two sequences is calculated are one-to-one corresponding. Therefore, before using the Euclidean distance algorithm to measure the similarity between the character sequence data set and the historical character sequence data set corresponding to the historical time series curve, it is necessary to obtain the historical sequence data set corresponding to each historical time series curve, and discretize the historical sequence data set based on the above-mentioned symbol clustering method to obtain the historical character sequence data set.

[0114] Exemplarily, after obtaining the historical character sequence data set corresponding to each historical time series curve, the following (a) to (c) may be included:

[0115] (a) determining the Euclidean distance between the character sequence data set and each of the historical character sequence data sets based on the Euclidean distance calculation formula; and determining the similarity between the character sequence data set and each of the historical character sequence data sets according to the calculation result;

[0116] (b) determining a plurality of similar time series curves corresponding to the character sequence dataset based on the similarity between the character sequence dataset and each of the historical character sequence datasets;

[0117] (c) performing curve weighting processing on the plurality of similar timing curves according to a preset curve adjustment method to obtain the target timing curve.

[0118] Here, the Euclidean distance calculation formula includes:

[0119]

[0120] Among them, L q,p represents the Euclidean value between the character sequence dataset q and the historical character sequence dataset p; q i represents the i-th data in the character sequence data set q; p i Represents the i-th data in the historical character sequence dataset p.

[0121] It can be understood that after calculating the Euclidean value between the character sequence dataset and each historical character sequence dataset, the similarity between the character sequence dataset and each historical character sequence dataset is calculated according to the calculation result, and the formula is expressed as follows:

[0122] S(Q,P)=[L q,p ] -1 ;

[0123] Among them, S(Q,P) represents the similarity between the character sequence dataset q and the historical character sequence dataset p.

[0124] Specifically, after obtaining the similarity between the character sequence data set and each historical character sequence data set, by sorting the values ​​of each similarity, multiple similar timing curves corresponding to the character sequence data set (i.e., historical timing curves corresponding to several historical character sequence data sets with higher similarity rankings) can be determined. At the same time, after obtaining multiple similar timing curves, a weighted adjustment method is used to fit multiple similar timing curves to obtain a target timing curve. Here, the weighted adjustment formula is:

[0125]

[0126] in, It is represented as the target time series data set corresponding to the target time series curve; represents the i-th data in the target time series data set; w represents the number of similar time series curves; a ji represents the i-th data of the historical sequence data set corresponding to the j-th similar time series curve; s j represents the similarity between the historical character sequence data set and the character sequence data set corresponding to the jth similar time series curve; s k Indicates the similarity between the historical character sequence dataset and the character sequence dataset corresponding to the kth similar time series curve.

[0127] S103, comparing the target time series curve with the time series data to be cleaned in the time series data set to be cleaned to determine abnormal data points; and correcting the time series data corresponding to the abnormal data points based on the target time series curve to obtain the target time series data set.

[0128] Here, since the change of power load data presents periodic characteristics and the value of power load is maintained within a specific threshold range, it is possible to determine whether the data is abnormal data by the average load value calculated in a fixed time period in the power system. In the present disclosure, the improved maximum threshold processing method is used to determine whether a load value is an abnormal value. Compared with the traditional maximum threshold method (using the average value of similar days), the load average value of the target time series curve is more accurate.

[0129] Specifically, the average value of the data in a fixed time period (fifteen days or thirty days) corresponding to the target time series curve is calculated, and the calculation result is used as the judgment value. Then, it is judged whether there are missing values ​​in each time series data to be cleaned in the time series data set to be cleaned. If there are, the time series data at the location of the missing value in the time series data set to be cleaned is completed based on the judgment value. Then, the abnormal data judgment rule is determined according to the judgment value, and it is judged whether each time series data to be cleaned in the time series data set to be cleaned meets the abnormal data judgment rule. If not, the time series data to be cleaned is an abnormal data point, and the time series data at the location of the abnormal data point in the time series data set to be cleaned is corrected based on the judgment value.

[0130] Among them, the abnormal data judgment rules include:

[0131]

[0132] in, Indicates the judgment value; x k Represents the kth time series data to be cleaned in the time series data set to be cleaned; Indicates the judgment threshold of the kth data; a k1 ,a k2 ,...,a kw Represents the kth data in the w historical character sequence data set.

[0133] In this way, after performing the above operation on each missing or abnormal data point in the time series data set to be cleaned, the cleaned target time series data set can be obtained.

[0134] The data cleaning method, device, medium and equipment based on time series similarity provided in the embodiments of the present disclosure reduce the complexity of data calculation by discretizing and reducing the dimension of the power load data to be cleaned using the symbol clustering method; and use the data after the dimension reduction process to measure the similarity with the historical character sequence data set, and then use the measurement result to compare with the original power load data set to determine the abnormal data points and correct them. In this way, the data cleaning method proposed in the present disclosure not only improves the accuracy and efficiency of power load data cleaning, but also effectively reduces the cost of manual operation.

[0135] Combine the following Figure 3 , the overall process of the data cleaning method based on time series similarity in the embodiment of the present disclosure is described in detail. Figure 3As shown in, first, a time series data set to be cleaned including time series data to be cleaned at different times in the power system is obtained, and then it is symbolically represented to obtain a character sequence data set including multiple character sequence data; secondly, similarity measurement is performed on the symbol sequence data set and multiple historical character sequence data sets respectively to determine the target time series curve; then based on the target time series curve, it is judged whether there are missing values ​​and abnormal data points in the time series data set to be cleaned. If so, the data at the corresponding position of the missing values ​​or abnormal data points is replaced by the average value of the target time series curve to obtain the target time series data set after data cleaning; if not, the target time series data set after data cleaning is directly obtained.

[0136] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0137] Based on the same inventive concept, the embodiment of the present disclosure also provides a data cleaning device based on time series similarity corresponding to the data cleaning method based on time series similarity. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned data cleaning method based on time series similarity in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the party, and the repeated parts will not be repeated.

[0138] Reference Figure 4 FIG. 4 is a schematic diagram of a data cleaning device 400 based on time series similarity provided by an embodiment of the present disclosure, wherein the device comprises:

[0139] The discretization processing module 401 is used to obtain a time series data set to be cleaned and a plurality of historical time series curves, wherein the time series data set to be cleaned includes time series data to be cleaned at different times in the power system; and discretize the time series data to be cleaned based on a symbol clustering method to obtain a character sequence data set, wherein the character sequence data set includes a plurality of character sequence data;

[0140] A similarity measurement module 402 is used to measure the similarity between the character sequence data set and the historical character sequence data set corresponding to the historical timing curve using a Euclidean distance algorithm to determine a target timing curve;

[0141] The data correction module 403 is used to compare the target time series curve with the time series data to be cleaned in the time series data set to be cleaned to determine abnormal data points; and correct the time series data corresponding to the abnormal data points based on the target time series curve to obtain the target time series data set.

[0142] In some possible embodiments, the discretization processing module 401 is further used for:

[0143] Acquire a power load statistical data set to be cleaned; the power load statistical data set includes a power load time series data set corresponding to a plurality of preset variables;

[0144] Based on the power load statistical data set, the maximum information coefficient between each preset variable and the power load is calculated as the correlation between each preset variable and the power load, the target variable among the preset variables is determined based on the correlation, and the power load time series data set corresponding to the target variable is obtained as the time series data set to be cleaned.

[0145] In some possible embodiments, the discretization processing module 401 is specifically used for:

[0146] The time series data to be cleaned are respectively normalized according to the data normalization formula, and the normalized data are respectively reduced in dimension based on the dimension reduction formula to obtain a reduced-dimension sequence data set; in the reduced-dimension sequence data set, the normalized data are distributed into a plurality of data intervals;

[0147] Performing symbol clustering processing on the reduced dimension sequence data set based on a preset symbol clustering standard to obtain the character sequence data set; the preset symbol clustering standard includes representing data in the same data interval in the reduced dimension sequence data set with the same character;

[0148] The data normalization formula includes:

[0149]

[0150] Among them, c i represents the i-th data in the dimensionality reduction sequence data set; x i represents the i-th data in the time series data set to be cleaned; μ(X) represents the average value of all the time series data to be cleaned in the time series data set to be cleaned; δ represents the standard deviation of all the time series data to be cleaned in the time series data set to be cleaned;

[0151] The dimension reduction processing formula includes:

[0152]

[0153] in, represents the i-th data of the character sequence data set; n represents the data length of the time series data set to be cleaned; N represents the number of segments represented by the dimension-reduced sequence data set; c j Represents the i-th data of the character sequence data set.

[0154] In some possible embodiments, the similarity measurement module 402 is further configured to:

[0155] A historical sequence data set corresponding to each historical time series curve is obtained, and the historical sequence data sets are discretized based on the symbol clustering method to obtain the historical character sequence data set.

[0156] In some possible embodiments, the similarity measurement module 402 is specifically used to:

[0157] Determine the Euclidean values ​​between the character sequence data set and each of the historical character sequence data sets based on the Euclidean distance calculation formula; and determine the similarity between the character sequence data set and each of the historical character sequence data sets according to the calculation result;

[0158] Determining a plurality of similar time series curves corresponding to the character sequence data set based on the similarity between the character sequence data set and each of the historical character sequence data sets;

[0159] Performing curve weighting processing on the plurality of similar timing curves according to a preset curve adjustment method to obtain a target timing curve;

[0160] The Euclidean distance calculation formula includes:

[0161]

[0162] Among them, L q,p represents the Euclidean value between the character sequence dataset q and the historical character sequence dataset p; q i represents the i-th data in the character sequence data set q; p i Represents the i-th data in the historical character sequence dataset p.

[0163] In some possible embodiments, the data correction module 403 is specifically used to:

[0164] Calculate the average value of the data in a fixed time period corresponding to the target time series curve, and use the calculation result as the judgment value;

[0165] Determine an abnormal data judgment rule according to the judgment value, and respectively judge whether each time series data to be cleaned in the time series data set to be cleaned meets the abnormal data judgment rule, if not, the time series data to be cleaned is an abnormal data point;

[0166] The abnormal data judgment rules include:

[0167]

[0168] Among them, x represents the judgment value; xk Represents the kth time series data to be cleaned in the time series data set to be cleaned; Indicates the judgment threshold of the kth data; a k1 ,a k2 ,...,a kw Represents the kth data in the w historical character sequence data set.

[0169] In some possible embodiments, the data correction module 403 is further used to:

[0170] Determine whether there are missing values ​​in each time series data to be cleaned in the time series data set to be cleaned, and if so, complete the time series data at the location of the missing value in the time series data set to be cleaned based on the determination value;

[0171] Accordingly, the data correction module 403 is specifically used for:

[0172] The time series data at the location where the abnormal data point is located in the time series data set to be cleaned is corrected based on the judgment value.

[0173] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Figure 5 , which is a schematic diagram of the structure of a computer device 500 provided in an embodiment of the present disclosure, including a processor 501, a memory 502, and a bus 503. The memory 502 is used to store execution instructions, including a memory 5021 and an external memory 5022; the memory 5021 is also called an internal memory, which is used to temporarily store the operation data in the processor 501 and the data exchanged with the external memory 5022 such as a hard disk. The processor 501 exchanges data with the external memory 5022 through the memory 5021.

[0174] In the embodiment of the present application, the memory 502 is specifically used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 501. That is, when the computer device 500 is running, the processor 501 communicates with the memory 502 through the bus 503, so that the processor 501 executes the application code stored in the memory 502, and then executes the method described in any of the above embodiments.

[0175] Among them, the memory 502 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0176] Processor 501 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0177] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the computer device 500. In other embodiments of the present application, the computer device 500 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0178] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data cleaning method based on time series similarity described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0179] The present disclosure also provides a computer program product, which carries a program code, and the instructions included in the program code can be used to execute the steps of the data cleaning method based on time series similarity described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here. Among them, the above computer program product can be implemented specifically through hardware, software or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium. In another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.

[0180] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0181] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0183] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0184] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A data cleaning method based on time series similarity, characterized in that: include: Acquire a time series data set to be cleaned and a plurality of historical time series curves, wherein the time series data set to be cleaned includes time series data to be cleaned at different times in the power system; and discretizing the time series data to be cleaned based on a symbol clustering method to obtain a character sequence data set, wherein the character sequence data set includes a plurality of character sequence data; Using the Euclidean distance algorithm to measure the similarity between the character sequence data set and the historical character sequence data set corresponding to the historical timing curve, and determine the target timing curve; The target time series curve is compared with the time series data to be cleaned in the time series data set to be cleaned to determine abnormal data points; and the time series data corresponding to the abnormal data points are corrected based on the target time series curve to obtain the target time series data set.

2. The method according to claim 1, characterized in that Before obtaining the time series data set to be cleaned and the multiple historical time series curves, the following steps are included: Acquire a power load statistical data set to be cleaned; the power load statistical data set includes a power load time series data set corresponding to a plurality of preset variables; Based on the power load statistical data set, the maximum information coefficient between each preset variable and the power load is calculated as the correlation between each preset variable and the power load, the target variable among the preset variables is determined based on the correlation, and the power load time series data set corresponding to the target variable is obtained as the time series data set to be cleaned.

3. The method according to claim 1, characterized in that The discretization processing of the time series data to be cleaned is performed based on the symbolic clustering method, including: The time series data to be cleaned are respectively normalized according to the data normalization formula, and the normalized data are respectively reduced in dimension based on the dimension reduction formula to obtain a reduced-dimension sequence data set; in the reduced-dimension sequence data set, the normalized data are distributed into a plurality of data intervals; Performing symbol clustering processing on the reduced dimension sequence data set based on a preset symbol clustering standard to obtain the character sequence data set; the preset symbol clustering standard includes representing data in the same data interval in the reduced dimension sequence data set with the same character; The data normalization formula includes: Among them, c i represents the i-th data in the dimensionality reduction sequence data set; x i represents the i-th data in the time series data set to be cleaned; μ(X) represents the average value of all the time series data to be cleaned in the time series data set to be cleaned; δ represents the standard deviation of all the time series data to be cleaned in the time series data set to be cleaned; The dimension reduction processing formula includes: in, represents the i-th data of the character sequence data set; n represents the data length of the time series data set to be cleaned; N represents the number of segments represented by the dimension-reduced sequence data set; c j Represents the i-th data of the character sequence data set.

4. The method according to claim 1, characterized in that: Before using the Euclidean distance algorithm to measure the similarity between the character sequence data set and the historical character sequence data set corresponding to the historical time series curve, the method includes: A historical sequence data set corresponding to each historical time series curve is obtained, and the historical sequence data sets are discretized based on the symbol clustering method to obtain the historical character sequence data set.

5. The method according to claim 4, characterized in that The method of using the Euclidean distance algorithm to measure the similarity between the character sequence data set and the historical character sequence data set corresponding to the historical timing curve to determine the target timing curve includes: Determine the Euclidean values ​​between the character sequence data set and each of the historical character sequence data sets based on the Euclidean distance calculation formula; and determine the similarity between the character sequence data set and each of the historical character sequence data sets according to the calculation result; Determining a plurality of similar time series curves corresponding to the character sequence data set based on the similarity between the character sequence data set and each of the historical character sequence data sets; Performing curve weighting processing on the plurality of similar timing curves according to a preset curve adjustment method to obtain a target timing curve; The Euclidean distance calculation formula includes: Among them, L q,p represents the Euclidean value between the character sequence dataset q and the historical character sequence dataset p; q i represents the i-th data in the character sequence data set q; p i Represents the i-th data in the historical character sequence dataset p.

6. The method according to claim 1, characterized in that The step of comparing the target time series curve with the time series data to be cleaned in the time series data set to be cleaned to determine abnormal data points includes: Calculate the average value of the data in a fixed time period corresponding to the target time series curve, and use the calculation result as the judgment value; Determine an abnormal data judgment rule according to the judgment value, and respectively judge whether each time series data to be cleaned in the time series data set to be cleaned meets the abnormal data judgment rule, if not, the time series data to be cleaned is an abnormal data point; The abnormal data judgment rules include: Among them, x represents the judgment value; x k Represents the kth time series data to be cleaned in the time series data set to be cleaned; represents the judgment threshold of the kth data; a k1 ,a k2 ,...,a kw Represents the kth data in the w historical character sequence data set.

7. The method according to claim 6, characterized in that Before respectively judging whether each time series data to be cleaned in the time series data set to be cleaned satisfies the abnormal data judgment rule, the method further includes: Determine whether there are missing values ​​in each time series data to be cleaned in the time series data set to be cleaned, and if so, complete the time series data at the location of the missing value in the time series data set to be cleaned based on the determination value; Accordingly, the method of correcting the time series data corresponding to the abnormal data point based on the target time series curve includes: The time series data at the location where the abnormal data point is located in the time series data set to be cleaned is corrected based on the judgment value.

8. A data cleaning device based on time series similarity, characterized in that: include: A discretization processing module is used to obtain a time series data set to be cleaned and a plurality of historical time series curves, wherein the time series data set to be cleaned includes time series data to be cleaned at different times in the power system; and discretize the time series data to be cleaned based on a symbol clustering method to obtain a character sequence data set, wherein the character sequence data set includes a plurality of character sequence data; A similarity measurement module, used to measure the similarity between the character sequence data set and the historical character sequence data set corresponding to the historical timing curve using the Euclidean distance algorithm to determine the target timing curve; The data correction module is used to compare the target time series curve with the time series data to be cleaned in the time series data set to be cleaned to determine abnormal data points; and based on the target time series curve, the time series data corresponding to the abnormal data points are corrected to obtain the target time series data set.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images