A method and apparatus for cleaning time series data

Abstract

This invention provides a time series data cleaning method and apparatus. The method includes: acquiring initial sequence data of an initial dataset, the initial sequence data including trend sequence data, periodic sequence data, and residual sequence data; performing a periodic component rationality judgment based on the initial sequence data to obtain target residual sequence data; performing anomaly detection on the target residual sequence data to obtain an outlier index of the initial dataset and performing an outlier removal operation to obtain the target dataset; and performing missing value imputation on the target dataset to complete the initial dataset cleaning. This invention uses a periodic component rationality judgment to analyze whether the periodic components of the sequence data are accurate and optimizes the residual sequence data. This significantly reduces the complexity of time series data while improving the accuracy of outlier identification during data cleaning, thus improving the accuracy, efficiency, and reliability of data cleaning, resulting in cleaner, more complete, and usable data after cleaning.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and apparatus for cleaning time series data. Background Technology

[0002] Current time series data anomaly detection models can be mainly divided into traditional statistical methods, represented by the 3σ criterion, box plots, and Grubbs hypothesis testing; and machine learning methods, represented by distance-based nearest neighbor algorithms, density-based local outlier detection algorithms, clustering-based density clustering algorithms, tree-based isolated forest algorithms, dimensionality reduction-based principal component analysis algorithms, classification-based support vector machines, and prediction-based difference ensemble moving average autoregressive models; in addition, there are deep learning methods, represented by feature extraction, learning normal feature representations, and end-to-end learning of anomaly scores.

[0003] While individual statistical models are low in complexity, fast in computation, and have strong generalization ability, their accuracy is generally low. Machine learning models, on the other hand, are more robust and have higher accuracy, but require training and generally have lower generalization ability. Furthermore, deep learning models, similar to machine learning models, suffer from unstable performance and high computational cost in current real-world business scenarios. Summary of the Invention

[0004] This invention provides a time series data cleaning method and apparatus to address the shortcomings of existing technologies in cleaning time series data, such as low efficiency and poor stability. It improves the accuracy, efficiency, and reliability of time series data cleaning, thereby making the cleaned data cleaner, more complete, and more usable.

[0005] In a first aspect, the present invention provides a time series data cleaning method, characterized in that it includes:

[0006] Obtain the initial sequence data of the initial dataset, which includes trend sequence data, periodic sequence data, and residual sequence data;

[0007] Obtain the proportion of the residual sequence data in the set of periodic sequence data and residual sequence data, and determine the target residual sequence data based on the proportion.

[0008] Anomaly detection is performed on the target residual sequence data to obtain the outlier index of the initial dataset and perform a removal operation to obtain the target dataset.

[0009] Furthermore, determining the target residual sequence data based on the proportion value includes the following steps:

[0010] When the proportion is less than a preset threshold, trend sequence data and periodic sequence data are removed from the initial dataset to obtain the target residual sequence data;

[0011] When the percentage value is greater than a preset threshold, the trend sequence data is removed from the initial dataset to obtain the target residual sequence data.

[0012] Furthermore, the step of performing anomaly detection on the target residual sequence data to obtain the target dataset includes the following steps:

[0013] Anomaly detection is performed on the target residual sequence data to obtain an intermediate dataset, which is a dataset after anomaly removal of outliers.

[0014] The intermediate dataset is interpolated to obtain the target dataset.

[0015] Furthermore, the step of performing data interpolation on the intermediate dataset to obtain the target dataset includes the following steps:

[0016] Obtain the length of consecutive missing values ​​in the intermediate dataset, and compare the length of the consecutive missing values ​​with a preset missing threshold;

[0017] When the length of the consecutive missing values ​​is less than the preset missing threshold, the consecutive missing values ​​are imputed using the first imputation method to obtain the target dataset.

[0018] When the length of the consecutive missing values ​​is greater than the preset missing threshold, the consecutive missing values ​​are interpolated using a second interpolation method to obtain the target dataset.

[0019] Furthermore, the step of performing anomaly detection on the target residual sequence data to obtain an intermediate dataset includes:

[0020] Anomaly detection is performed on the target residual sequence data based on the confidence interval of the target residual sequence data;

[0021] Identify outliers in the target residual sequence data that exceed the confidence interval;

[0022] Remove the outliers to obtain an intermediate dataset.

[0023] Furthermore, the first interpolation method is linear interpolation, and the second interpolation method is piecewise cubic interpolation.

[0024] Furthermore, obtaining the initial sequence data of the initial dataset includes the following steps:

[0025] Based on the seasonal decomposition algorithm, a multiplicative model is used to perform time-series decomposition on the initial dataset to obtain the initial sequence data.

[0026] Furthermore, the steps for obtaining the initial dataset include:

[0027] Obtain the original detection dataset, which includes at least one column of time series data and one column of sequence data to be cleaned;

[0028] A set of missing values ​​is obtained by performing preliminary missing value detection on the sequence data to be cleaned in the original detection dataset.

[0029] The missing value set is initially filled using linear interpolation to obtain an initial dataset;

[0030] The preliminary detection of missing values ​​includes: obtaining the percentage of missing values ​​and the location data of the missing values.

[0031] Secondly, the present invention also provides a time series data cleaning apparatus, characterized in that it comprises:

[0032] The acquisition module is used to acquire the initial sequence data of the initial dataset, which includes trend sequence data, periodic sequence data and residual sequence data;

[0033] The judgment and processing module is used to obtain the proportion of the residual sequence data in the set of periodic sequence data and residual sequence data, and determine the target residual sequence data based on the proportion.

[0034] An anomaly detection module is used to perform anomaly detection on the target residual sequence data, obtain the outlier index of the initial dataset, and perform a removal operation to obtain the target dataset.

[0035] Thirdly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs any of the steps described above.

[0036] (1) The time series data cleaning method and apparatus provided by the present invention obtains the sequence data of the data to be cleaned by performing time series decomposition on the data to be cleaned, and uses the periodic component rationality judgment to judge whether the periodic component of the sequence data is accurate, and optimizes the residual sequence data. In this way, the accuracy of outlier identification during data cleaning is improved on the basis of significantly reducing the complexity of time series data, and the accuracy, efficiency and reliability of data cleaning are improved, so that the cleaned data is cleaner, more complete and usable.

[0037] (2) By using only one parameter for optimization, the present invention makes the training cost of the data cleaning model obtained by the invention method low, the model optimization difficulty easy, and the model anomaly detection efficiency high.

[0038] (3) This invention combines and improves the data missing interpolation method. It uses the number of consecutive missing items and a consecutive missing item threshold for judgment. Within the threshold range, linear interpolation is sufficient to meet the requirements. Outside the threshold range, a piecewise cubic interpolation algorithm is used for missing item processing. Using combined missing item interpolation can effectively reduce computational resource consumption and improve missing item interpolation efficiency. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0040] Figure 1 This is one of the flowcharts illustrating the time series data cleaning method provided by the present invention;

[0041] Figure 2 This is a flowchart illustrating the data interpolation method provided by the present invention;

[0042] Figure 3 This is the second flowchart illustrating the time series data cleaning method provided by the present invention;

[0043] Figure 4 This invention provides monitoring data and a schematic diagram of detected anomalies and actual anomalies.

[0044] Figure 5 This is a schematic diagram of the time series data cleaning device provided by the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0046] It should be noted that, in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0047] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more.

[0048] Time series data is a series of data points arranged in chronological order. Each data point represents a numerical value of a phenomenon observed at a given point in time. These data points are typically sampled at fixed time intervals, such as per second, per minute, per hour, or per day.

[0049] The following is combined with Figures 1-5 This invention describes the time series data cleaning method and apparatus provided in the embodiments of the present invention.

[0050] Figure 1 This is one of the flowcharts illustrating the time series data cleaning method provided by the present invention, such as... Figure 1 As shown, including but not limited to the following steps:

[0051] S110: Obtain the initial sequence data of the initial dataset, which includes trend sequence data, periodic sequence data and residual sequence data.

[0052] Specifically, the initial dataset is decomposed into time series using a seasonal decomposition algorithm, and the three series data of trend component T, periodic component S, and residual component R are obtained sequentially. The periodic component is a new sequence composed of points with periodic variation patterns in the time series data. It should be noted that in the embodiments of the present invention, trend component T refers to trend series data, periodic component S refers to periodic series data, and residual component R refers to residual series data.

[0053] S120: Obtain the proportion of the residual sequence data in the set of periodic sequence data and residual sequence data, and determine the target residual sequence data based on the proportion.

[0054] Specifically, based on the above three types of sequence data, a reasonableness judgment of the periodic component is added to improve the residual sequence data composed of residual vectors. The core implementation idea is to first calculate the proportion of the residual sequence data in the set of periodic sequence data and residual sequence data, and then select the optimization method for the residual sequence data based on the proportion data. The step of determining the target residual sequence data according to the proportion value includes the following steps:

[0055] When the proportion is less than a preset threshold, trend sequence data and periodic sequence data are removed from the initial dataset to obtain the target residual sequence data;

[0056] When the percentage value is greater than a preset threshold, the trend sequence data is removed from the initial dataset to obtain the target residual sequence data.

[0057] In this embodiment, the proportion of the residual sequence data R in (S+R) is first calculated. If the proportion is less than a preset threshold, such as 0.4, it indicates that the remaining part of the sequence after removing the trend component still has strong random fluctuations, meaning that the periodic component of the sequence is considered non-negligible. The target residual sequence data is then determined to be the result of removing both the trend component and the periodic component from the initial dataset. If the proportion is greater than the preset threshold, such as 0.4, the periodic component is considered to be a regular change and can be excluded from anomaly detection as a characteristic sequence of the initial dataset. The target residual sequence data is then determined to be the difference between the initial dataset and the trend sequence data. That is, when R / (S+R) is less than 0.4, R = Y1 - TS; otherwise, R = Y1 - T.

[0058] S130: Perform anomaly detection on the target residual sequence data, obtain the outlier index of the initial dataset, and perform a removal operation to obtain the target dataset.

[0059] In this embodiment, anomaly detection is performed on the target residual sequence data, and then the positions of abnormal data in the initial dataset are selected to form an outlier index. Data located at the outlier index is removed. Based on this, the target dataset after anomaly processing is obtained.

[0060] This invention obtains the sequence data of the data to be cleaned by performing time-series decomposition on the data to be cleaned, and uses the rationality judgment of periodic components to determine whether the periodic components of the sequence data are accurate, and optimizes the residual sequence data. In this way, the complexity of time-series data is significantly reduced, the accuracy of outlier identification during data cleaning is improved, and the accuracy, efficiency and reliability of data cleaning are improved, so that the cleaned data is cleaner, more complete and usable.

[0061] Based on the above embodiments, as an optional embodiment, the time series data cleaning method provided by the present invention includes the following steps for obtaining the initial sequence data of the initial dataset: using a multiplicative model, the initial dataset is decomposed into time series using a moving average method to obtain the initial sequence data.

[0062] In this embodiment, based on the improved seasonal decomposition algorithm, a multiplicative model is used to perform time-series decomposition of the Val column of the initial dataset Y1 using a moving average method. The specific operation of the improved seasonal decomposition algorithm is as follows:

[0063] (1) Determine the time series period m: Obtain the time interval of the time column of the initial dataset Y1, and convert this time interval into the seasonal variation period value m. The conversion relationship between the time interval and the period is shown in Table 1.

[0064] Table 1. Correspondence between time intervals and seasonal variation cycles.

[0065]

[0066] (2) Determine the trend component T: Based on the determined seasonal variation cycle value m, the trend component T is calculated using the moving average method.

[0067] When m is an odd number, the moving average method uses the following formula:

[0068]

[0069] When m is an even number, the moving average method uses the following formula:

[0070]

[0071] Among them, T t t represents the smoothed value corresponding to time point t; f represents the time series frequency value; m represents the time series period value, i.e., the number of measured values ​​for each moving average; Y represents the original hydrological monitoring data.

[0072] (3) Determine the periodic component S: Average the values ​​of the same frequency within each period to obtain the periodic component s. Average the periodic components s to obtain the centered periodic component S. The relationship between the trend component T, the periodic component S, the residual component R, and the original data in the Val column is as follows:

[0073] Val t =T t ×S t ×R t

[0074] The formula for calculating the periodic component s is:

[0075]

[0076] Where n is the number of time series frequency values ​​in time series period m, n = max(n, nf ≤ m); f is the time series frequency value.

[0077] Furthermore, based on the initial sequence data, the rationality of the periodic components is judged, and the target residual component R is determined: that is, the proportion of R in (S+R) is calculated. If the proportion is less than 0.4, then R... t =Val t -T t -S t Otherwise, R t =Val t -T t .

[0078] This invention performs time-series decomposition of raw data based on a seasonal decomposition algorithm. However, considering the assumption that the seasonal decomposition algorithm assumes each periodic component is identical, which is unreasonable for actual hydrological data, this invention improves the residual vector of the seasonal decomposition algorithm by adding a periodic component rationality judgment. Based on this improvement, anomaly detection is then implemented. Therefore, the time-series data cleaning model established using the proposed method has low training costs, easy model tuning, high anomaly detection efficiency, and high robustness and detection accuracy.

[0079] Based on the above embodiments, as an optional embodiment, the time series data cleaning method provided by the present invention includes the following steps: performing anomaly detection on the target residual sequence data to obtain a target dataset: performing anomaly detection on the target residual sequence data to obtain an intermediate dataset, wherein the intermediate dataset is a dataset after anomaly removal of outliers, and performing data interpolation on the intermediate dataset to obtain the target dataset.

[0080] Specifically, a confidence interval is set for the target residual sequence data, anomaly detection is performed based on the confidence interval, and the positions of outliers in the target residual sequence data that exceed the confidence interval are identified; data located at the outlier positions are removed to obtain an intermediate dataset.

[0081] In this embodiment, referring to the anomaly detection principle of box plots, anomaly detection is performed on the decomposed residual sequence data R by setting confidence intervals. Specifically, the lower quartile Q of the residual sequence R is calculated. 0.25 and the upper quartile Q 0.75 And use the following formula:

[0082] Q down_limit =Q 0.25 +k(Q 0.75 -Q 0.25 )

[0083] Q up_limit =Q 0.75 +k(Q 0.75 -Q 0.25 )

[0084] The lower and upper bounds of the confidence intervals are calculated separately, the interval proportion parameter k is optimized, and values ​​exceeding the upper and lower bounds in the residual sequence R are marked as outliers, thus obtaining the outlier index of the initial dataset. Based on historical data, the parameter k value is updated, and the optimal k value corresponding to the combined optimal anomaly detection hit rate, false negative rate, and false positive rate is obtained, completing the anomaly detection model calibration. Table 2 shows the quartiles of the target residual sequence R obtained from the initial dataset to be cleaned.

[0085] Table 2. Quartiles corresponding to residual components

[0086]

[0087] It should be noted that, based on the relationship between the trend component T, the periodic component S, and the residual component R and the original data in the Val column, it can be seen that the outlier positions of the initial dataset Y1 are consistent with the outlier positions of the residual sequence R. That is, the outlier position after the anomaly detection of the residual sequence is the outlier index position of the original data.

[0088] Furthermore, based on the outlier index, data at outlier locations in the initial dataset Y1 are directly removed. Based on this, the intermediate dataset Y2 after anomaly processing is obtained, and the anomaly detection model's performance metrics are shown in Table 3.

[0089] Table 3. Anomaly Detection Status of Original Data

[0090]

[0091] Based on the above embodiments, as an optional embodiment, the time series data cleaning method provided by the present invention includes the following steps for obtaining the initial dataset: obtaining an original detection dataset, wherein the original detection dataset includes at least one column of time series data and one column of sequence data to be cleaned; performing preliminary missing value detection on the sequence data to be cleaned in the original detection dataset to obtain a set of missing values; performing preliminary filling of the missing values ​​in the original detection dataset using linear interpolation to obtain an initial dataset; the preliminary missing value detection includes: obtaining the percentage data of missing values ​​and the position data corresponding to the missing values.

[0092] In this embodiment, firstly, the original monitoring dataset Y0 is obtained, which contains at least one time series column (time column) and one sequence to be cleaned column (Val column). Preliminary missing value detection is performed on the Val column of the original monitoring dataset Y0, the percentage of missing values ​​and their corresponding positions are calculated, and preliminary filling of missing values ​​is performed using linear interpolation. Based on this, a new initial dataset Y1 is formed.

[0093] Based on the content of the above embodiments, as an optional embodiment, Figure 2 This is a flowchart illustrating the data interpolation method provided by the present invention, as shown below. Figure 2 As shown, the time series data cleaning method provided by the present invention includes the following steps for imputing data in the intermediate dataset to obtain the target dataset: obtaining the length value of consecutive missing values ​​in the intermediate dataset, and comparing the length value of the consecutive missing values ​​with a preset missing threshold.

[0094] Specifically, when the length of the consecutive missing values ​​is less than the preset missing threshold, the consecutive missing values ​​are imputed using a first imputation method to obtain the target dataset; when the length of the consecutive missing values ​​is greater than the preset missing threshold, the consecutive missing values ​​are imputed using a second imputation method to obtain the target dataset.

[0095] Optionally, the first interpolation method may be selected as linear interpolation or other interpolation methods with the same effect as the linear interpolation method, and the second interpolation method may be selected as piecewise cubic interpolation or other interpolation methods with the same effect as the piecewise cubic interpolation algorithm. Those skilled in the art can make adaptive adjustments as needed, and the present invention does not limit this.

[0096] In this embodiment, the number of consecutive missing values ​​Len in the intermediate dataset Y2 is counted, the length of consecutive missing values ​​in each missing part of the dataset is obtained, and the length of missing values ​​in each part is compared with a preset missing threshold. For missing parts with a missing value length less than the missing threshold, a first imputation method is used for data imputation, and for missing parts with a missing value length greater than the missing threshold, a second imputation method is used for data imputation.

[0097] Optionally, missing values ​​within a specified consecutive missing threshold (default, for example, 5) are imputed using linear interpolation. If the dataset still has missing parts after linear interpolation, piecewise cubic interpolation is used to impute the missing values, resulting in the final target dataset Y after data cleaning. The linear interpolation and piecewise cubic interpolation algorithms can be directly implemented using the interpolate function in Python, which is known to those skilled in the art, and will not be elaborated upon here.

[0098] Based on the above embodiments, as an optional embodiment, taking hydrological time series data as an example, a specific embodiment of using time series data cleaning methods to clean hydrological time series data is provided.

[0099] Hydrological and rainfall monitoring data are the most important hydrological time-series data for engineering projects. The collection and integration of hydrological data is a crucial foundation for the construction of hydrological databases and a prerequisite for the normal operation of all hydrological work. Anomalies in hydrological time-series data are typically caused by various factors such as sudden equipment failures, equipment upgrades, and adjustments to equipment monitoring benchmarks. These anomalies include missing data (omissions), local point anomalies (outliers), and ensemble anomalies in the hydrological data transmitted from the ground. The accuracy of hydrological data directly affects the reliability of research results based on it. For example, inaccurate monitoring of water level, rainfall, and flow information will lead to simulation results from hydrodynamic models, hydrological models, or big data prediction models that are difficult to match with actual hydrological changes. The model simulation accuracy will not meet actual production needs, making it difficult to provide real-time and accurate decision support for reservoir scheduling decisions, urban flood control and drainage, and other areas. Therefore, accurate hydrological data cleaning is of great significance for ensuring hydrological data quality and guaranteeing the use of subsequent model base data.

[0100] Figure 3 This is the second flowchart illustrating a time-series data cleaning method provided by the present invention, using hydrological time-series data as an example. Figure 3 As shown,

[0101] Step 1: Obtain the original monitoring dataset Y0. The data must include one time series column (time column) and one column of data to be cleaned (Val column). Areal rainfall and other correlated series data can be added between the time series and the data to be cleaned. The data interval is on an hourly scale.

[0102] Step 2: Perform preliminary missing value detection on the Val column of the original monitoring dataset Y0, calculate the percentage and location of missing values, and perform preliminary filling of missing values ​​using linear interpolation. Based on this, a new complete initial dataset Y1 is formed.

[0103] The Val column in the original data is linearly interpolated to form the initial dataset Y1. The Val sequence values ​​in Y1 are as follows: Figure 4 As shown in (a);

[0104] Step 3: Perform an improved seasonal decomposition on the initial dataset Y1 to obtain the sequence residual components R. Based on the improved seasonal decomposition algorithm, a multiplicative model is used to perform time-series decomposition on the Val column of the initial dataset Y1 using a moving average method. The specific time-series decomposition operation is as follows:

[0105] (1) Determine the time series period m: The detection interval of the initial dataset Y1 is on an hourly scale. According to Table 1, the time series period m is 24.

[0106] (2) The trend component T is calculated using the moving average method in the following formula.

[0107] When m is an odd number, the moving average method uses the following formula:

[0108]

[0109] When m is an even number, the moving average method uses the following formula:

[0110]

[0111] Among them, T t t represents the smoothed value corresponding to time point t; f represents the time series frequency value; m represents the time series period value, i.e., the number of measured values ​​for each moving average; Y represents the original hydrological monitoring data.

[0112] (3) The periodic component s is determined by the following formula, and the calculated s sequence value is averaged to obtain the periodic component S.

[0113] The relationship between the trend component T, the periodic component S, and the residual component R and the original data in the Val column is as follows:

[0114] Val t =T t ×S t ×R t

[0115] The formula for calculating the periodic component s is:

[0116]

[0117] Where n is the number of time series frequency values ​​in time series period m, n = max(n, nf ≤ m); f is the time series frequency value.

[0118] (4) Combining the relationship between the trend component T, the periodic component S, and the residual component R with the original data in the Val column, determine the reasonableness of the periodic component and determine the residual component R: that is, calculate the proportion of R in (S+R). If the proportion is less than 0.4, then R... t =Val t -T t -S t Otherwise R t =Val t -T t The calculated R value for the dataset is as follows: Figure 4 As shown in (b).

[0119] Step 4: Referring to the anomaly detection principle of box plots, anomaly detection is performed on the decomposed residual sequence data R by setting confidence intervals. Specifically, confidence intervals are set, model parameters are calibrated, and anomaly detection is performed on the residual components R.

[0120] Calculate the lower quartile Q corresponding to the residual component R. 0.25 and the upper quartile Q 0.75 The interval ratio parameter k is initialized to 3. Based on historical data, the value of parameter k is updated. The k value corresponding to the optimal time when the model's anomaly detection hit rate, false negative rate and false positive rate are obtained, and the anomaly detection model calibration is completed.

[0121] Specifically, calculate the lower quartile Q of the residual sequence R. 0.25 and the upper quartile Q 0.75 And use the following formula:

[0122] Q down_limit =Q 0.25 +k(Q 0.75 -Q 0.25 )

[0123] Q up_limit =Q 0.75 +k(Q 0.75 -Q 0.25 )

[0124] Calculate the lower and upper limits of the confidence intervals, optimize the interval ratio parameter k, mark the values ​​in the residual sequence R that exceed the upper and lower limits as outliers, and obtain the outlier index of the initial dataset.

[0125] It should be noted that, based on the relationship between the trend component T, the periodic component S, and the residual component R and the original data in the Val column, it can be seen that the outlier positions of the initial dataset Y1 are consistent with the outlier positions of the residual sequence R. That is, the outlier position after the anomaly detection of the residual sequence is the outlier index position of the original data.

[0126] Step 5: Based on the outlier index, remove outliers from the initial dataset Y1 to form the intermediate dataset Y2.

[0127] Based on the outlier index, data at outlier locations in the initial dataset Y1 are directly removed. This results in the intermediate dataset Y2 after outlier handling.

[0128] Step 6: Set the threshold for continuous missing data and select the missing data imputation method.

[0129] The number of consecutive missing values ​​Len in the intermediate dataset Y2 is counted. For missing values ​​within the specified consecutive missing threshold (default is 5), linear interpolation is used to impute the missing values. If missing values ​​still exist after linear interpolation, piecewise cubic interpolation is used to impute the missing values, resulting in the final target dataset Y after data cleaning.

[0130] Step 7: After interpolation, the data is processed to obtain the final cleaned target dataset Y.

[0131] This invention combines and improves data missing interpolation methods. It uses the number of consecutive missing values ​​and a consecutive missing value threshold for judgment. Within the threshold range, linear interpolation is sufficient; outside the threshold range, a piecewise cubic interpolation algorithm is used for missing value processing. This combined missing value interpolation effectively reduces computational resource consumption and improves interpolation efficiency.

[0132] Figure 5 This is a schematic diagram of the time series data cleaning device provided by the present invention, as shown below. Figure 5 As shown, the device includes: an acquisition module 501, a judgment and processing module 502, and an anomaly detection module 503.

[0133] The acquisition module 501 is used to acquire the initial sequence data of the initial dataset.

[0134] The initial sequence data includes trend sequence data, periodic sequence data, and residual sequence data.

[0135] The judgment and processing module 502 is used to obtain the proportion of the residual sequence data in the set of periodic sequence data and residual sequence data, and determine the target residual sequence data based on the proportion.

[0136] The anomaly detection module 503 is used to perform anomaly detection on the target residual sequence data, obtain the outlier index of the initial dataset, and perform a removal operation to obtain the target dataset.

[0137] It should be noted that the time series data cleaning apparatus provided in this embodiment of the invention can execute the time series data cleaning method described in any of the above embodiments during specific operation, and this embodiment will not elaborate on this.

[0138] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the time series data cleaning method provided in the above embodiments, the method including: obtaining initial sequence data of an initial dataset, the initial sequence data including trend sequence data, periodic sequence data and residual sequence data; performing a periodic component rationality judgment based on the initial sequence data to obtain target residual sequence data; the periodic component rationality judgment is by obtaining the proportion value of the residual sequence data in the set of periodic sequence data and residual sequence data, and determining the target residual sequence data according to the proportion value; performing anomaly detection on the target residual sequence data to obtain the outlier index of the initial dataset and performing a removal operation to obtain the target dataset; performing missing imputation on the target dataset to complete the initial dataset cleaning.

[0139] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the time-series data cleaning method provided in the above embodiments. The method includes: acquiring initial sequence data of an initial dataset, the initial sequence data including trend sequence data, periodic sequence data, and residual sequence data; performing a periodic component rationality judgment based on the initial sequence data to obtain target residual sequence data; the periodic component rationality judgment is performed by obtaining the proportion of the residual sequence data in the set of periodic sequence data and residual sequence data, and determining the target residual sequence data according to the proportion; performing anomaly detection on the target residual sequence data to obtain the outlier index of the initial dataset and performing a removal operation to obtain the target dataset; and performing missing imputation on the target dataset to complete the initial dataset cleaning.

[0140] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A time series data cleaning method, characterized in that, The method comprises the following steps: obtaining initial sequence data of an initial data set, wherein the initial sequence data comprises trend sequence data, periodic sequence data and residual sequence data; obtaining a proportion value of the residual sequence data in a combination of the periodic sequence data and the residual sequence data, and removing the trend sequence data and the periodic sequence data from the initial data set to obtain target residual sequence data when the proportion value is less than a preset threshold value; removing the trend sequence data from the initial data set to obtain the target residual sequence data when the proportion value is greater than the preset threshold value; performing anomaly detection on the target residual sequence data to obtain an intermediate data set, wherein the intermediate data set is a data set after removing abnormal values; performing data interpolation on the intermediate data set to obtain a target data set, wherein the performing data interpolation on the intermediate data set to obtain the target data set comprises the following steps: obtaining a length value of continuous missing values in the intermediate data set, and comparing the length value of the continuous missing values with a preset missing threshold value; when the length value of the continuous missing values is less than the preset missing threshold value, performing data interpolation on the continuous missing values by using a first interpolation method to obtain the target data set; when the length value of the continuous missing values is greater than the preset missing threshold value, performing data interpolation on the continuous missing values by using a second interpolation method to obtain the target data set, wherein the first interpolation method is a linear interpolation method, and the second interpolation method is a piecewise cubic interpolation algorithm. 2.The time series data cleaning method of claim 1, wherein, The performing anomaly detection on the target residual sequence data to obtain the intermediate data set comprises the following steps: performing anomaly detection on the target residual sequence data according to a confidence interval of the target residual sequence data; identifying abnormal values in the target residual sequence data that are out of the confidence interval; removing the abnormal values to obtain the intermediate data set. 3.The time series data cleaning method of claim 1, wherein, The obtaining initial sequence data of an initial data set comprises the following steps: performing time series decomposition on the initial data set by using a multiplication model according to a seasonal decomposition algorithm to obtain the initial sequence data. 4.The time series data cleaning method of claim 1, wherein, The obtaining step of the initial data set comprises the following steps: obtaining an original detection data set, wherein the original detection data set comprises at least one column of time sequence data and one column of to-be-cleaned sequence data; performing preliminary detection on missing values in the to-be-cleaned sequence data in the original detection data set to obtain a missing value set; performing preliminary filling on the missing value set by using a linear interpolation method to obtain the initial data set; the preliminary detection on the missing values comprises obtaining missing value proportion data and missing value corresponding position data.

5. A time series data cleaning apparatus characterized by comprising: The method comprises the following steps: an obtaining module is configured to obtain initial sequence data of an initial data set, wherein the initial sequence data comprises trend sequence data, periodic sequence data and residual sequence data; a judgment processing module is configured to obtain a proportion value of the residual sequence data in a combination of the periodic sequence data and the residual sequence data, and determine target residual sequence data according to the proportion value; remove the trend sequence data and the periodic sequence data from the initial data set to obtain the target residual sequence data when the proportion value is less than a preset threshold value; and remove the trend sequence data from the initial data set to obtain the target residual sequence data when the proportion value is greater than the preset threshold value. Anomaly detection module, used for anomaly detection on the target residual sequence data, obtaining an intermediate data set, the intermediate data set being a data set after anomaly elimination; The intermediate data set is subjected to data interpolation to obtain a target data set, and the data interpolation on the intermediate data set to obtain the target data set comprises the steps of: Obtaining a length value of continuous missing values in the intermediate data set, and comparing the length value of the continuous missing values with a preset missing threshold value; When the length value of the continuous missing values is less than the preset missing threshold value, a first interpolation method is used for data interpolation on the continuous missing values to obtain a target data set; When the length value of the continuous missing values is greater than the preset missing threshold value, a second interpolation method is used for data interpolation on the continuous missing values to obtain a target data set, the first interpolation method being a linear interpolation method, and the second interpolation method being a piecewise cubic interpolation algorithm.

6. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to realize the steps of the time series data cleaning method according to any one of claims 1 to 4.

Images