Time sequence detection and cleaning method and system for power abnormal data

Abstract

The invention provides a time sequence detection and cleaning method and system for power abnormal data. The method comprises the following steps: taking power time sequence data as input, and determining an electric signal prediction index at the current moment based on a time sequence correlation analysis model; determining an abnormal characteristic value based on the electric signal prediction index and actual monitoring data at the current moment, and determining parallel detection intensity based on the abnormal characteristic value; based on the parallel detection intensity, identifying first abnormal data and suspected abnormal data in the power time sequence data, and screening second abnormal data in the suspected abnormal data based on a target condition; and for respective corresponding data points of the first abnormal data and the second abnormal data in the time sequence, deleting the abnormal data, and performing data prediction and filling on the positions of the data points by using non-abnormal data in the set time window to which each data point belongs, so as to obtain cleaned power time sequence data. The invention provides a data detection and cleaning mode taking time sequence, precision and efficiency into consideration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric power data processing, in particular to a time sequence detection and cleaning method and system for electric power abnormal data. BACKGROUND

[0002] The electric power system is the core infrastructure of modern society, and its safe, stable and efficient operation highly depends on high-quality operation data. With the deepening of the construction of smart grids, especially the large-scale access of new energy, the enhancement of user-side interaction and the popularization of advanced measurement systems (AMI), the time series data of loads, voltages, currents and other data collected by the power grid has shown explosive growth, and the volatility and complexity have significantly increased. These data are the basis for a series of work such as state monitoring, load forecasting, fault diagnosis, energy efficiency analysis and market transactions. However, in the process of data collection, transmission and storage, abnormal data will inevitably be introduced due to equipment failure, communication interference, human error or extreme events (such as lightning).

[0003] Traditional detection methods based on fixed thresholds or simple statistical rules are difficult to adapt to the complex time sequence correlation, periodicity and nonlinearity of electric power data, resulting in high false positive and false negative rates. Low-quality data can seriously distort the analysis results, mislead the dispatching decisions, and even trigger a chain of failures.

[0004] Therefore, the present application provides a time sequence detection and cleaning method and system for electric power abnormal data to solve one of the above technical problems. SUMMARY

[0005] The purpose of the present application is to provide a time sequence detection and cleaning method and system for electric power abnormal data, which can solve at least one of the above technical problems. The specific scheme is as follows: According to the specific embodiment of the present application, in a first aspect, the present application provides a time sequence detection and cleaning method for electric power abnormal data, comprising: Taking electric power time sequence data as input, determining an electric signal prediction index of the current time based on a time sequence correlation analysis model; based on the electric signal prediction index and the actual monitoring data of the current time, determining an abnormal feature value representing the difference between the data and the conventional time sequence trend, and based on the abnormal feature value, determining a parallel detection strength representing the discrimination accuracy of the abnormal data; based on the parallel detection strength, identifying first abnormal data and suspected abnormal data in the electric power time sequence data, and based on a target condition, screening second abnormal data from the suspected abnormal data; for the data points corresponding to the first abnormal data and the second abnormal data in the time sequence respectively, deleting abnormal data, using non-abnormal data within a set time window to which each data point belongs to perform data prediction and filling at the position of the data point, and obtaining cleaned electric power time sequence data.

[0006] In one implementation, the time-series correlation analysis model is expressed by the following formula: ;in, for Time-series-based electrical signal prediction metrics and This represents the coding coefficients of two different power anomaly data. Corresponding to equipment failure type exceptions, Corresponding to communication interference anomalies, and They are respectively Coding coefficients at time 1 and The corresponding data sequence parameters, and They are coding coefficients and Corresponding big data prediction permissions for power anomalies and They are coding coefficients and The corresponding time-series trend function.

[0007] In one embodiment, the method further includes: determining discretization correction permissions based on the time-series correlation analysis model, wherein the discretization correction permissions are used for the separation of anti-interference and anomaly early warning; constructing a Hadoop parallel processing platform based on the discretization correction permissions; performing time-by-time parallel detection on the power time-series data based on the Hadoop parallel processing platform to initially separate the power time-series data into a non-abnormal data candidate set and an abnormal data candidate set; wherein the actual monitoring data at the current moment is selected from the abnormal data candidate set.

[0008] In one implementation, the discretization correction authority is determined using the following formula: ;in, Indicates the permission to modify discretization. This represents a predefined, actual, non-abnormal data. This represents the average of the actual non-abnormal data that the power control unit can record per unit of time. This represents the initial value of the discretization distribution coefficient for large-scale power anomaly data. Represent a Big data on power anomalies with defined time points. This represents the adjustment ratio coefficient for electricity data based on a time series forecasting model. This indicates the calibration characteristics for correcting given electricity data; among which, The dimensionality reduction prediction coefficients are calculated using the following formula: ;in, Represents the dimensionality reduction prediction coefficient. and This represents two randomly selected scaling parameters for large datasets of power anomalies. The minimum dimensionality reduction coefficient, and These are the prediction permission values ​​for actual non-abnormal data and large-scale abnormal power data, respectively. For time-series coordination coefficients; In one embodiment, the Hadoop parallel processing platform includes: an HBase component for storing structured power data, the structured power data including the actual non-abnormal data, the defined large data of power anomalies, and the discretized correction permissions; a Hive component for filtering the candidate set of non-abnormal data and the candidate set of abnormal data using SQL statements; and a Sqoop component for real-time acquisition of data at the current moment and synchronous transmission of historical databases.

[0009] In one implementation, the abnormal feature value is determined using the following formula: ;in, Indicates abnormal feature values, This represents the coefficient of abnormal behavior in power big data. This indicates the stable fluctuation frequency of abnormal power consumption indicators. This represents the unit fluctuation period of the electrical signal.

[0010] In one implementation, the parallel detection intensity is determined using the following formula: ;in, Indicates the parallel detection intensity. Indicates abnormal feature values The power signal transmission vector, This represents the given big data labeling coefficients for power anomalies. , ... express Different permissions for parallel transmission of big data on power anomalies This indicates the maximum prediction authority for power big data under time series conditions. This represents a predetermined power signal detection scaling parameter.

[0011] 8. The method according to claim 1, characterized in that, the step of identifying the first abnormal data and suspected abnormal data in the power time series data based on the parallel detection intensity includes: dividing the collected historical power data into a training set and a validation set, using the training set to train the time series correlation analysis model with the training objective of minimizing the mean square error between the power signal prediction index and the actual non-abnormal data; using the trained time series correlation analysis model to predict the power time series data time by time to obtain a baseline curve of the power signal prediction index changing over time; for the actual monitoring data at each time, if the abnormal feature value corresponding to the actual monitoring data is less than or equal to a first threshold, it is determined to be non-abnormal candidate data; if the abnormal feature value corresponding to the actual monitoring data is greater than the first threshold, and the parallel detection intensity corresponding to the actual monitoring data is greater than or equal to a second threshold calibrated by the validation set, it is determined to be the first abnormal data; if the abnormal feature value corresponding to the actual monitoring data is greater than the first threshold, and the parallel detection intensity corresponding to the actual monitoring data is less than the second threshold, it is determined to be the suspected abnormal data.

[0012] In one implementation, data prediction and filling are performed on the locations of the data points using non-abnormal data within a set time window to which each data point belongs. This includes: deleting data from the locations of each data point in the time series; constructing an ARIMA(p, d, q) model based on non-abnormal data within the set time windows before and after each data point; performing data prediction on the locations of each data point based on the ARIMA(p, d, q) model; calculating the temporal trend similarity for each filled data point according to the time window of its location; and filling and cleaning the data when the temporal trend similarity is greater than or equal to a set similarity threshold.

[0013] According to a specific embodiment of this application, in a second aspect, this application provides a time-series detection and cleaning system for power anomaly data, comprising: The system comprises the following modules: a prediction module, which takes power time-series data as input and determines the power signal prediction index for the current moment based on a time-series correlation analysis model; a feature calculation module, which determines abnormal feature values ​​representing the difference between the power signal prediction index and the actual monitoring data at the current moment, and determines the parallel detection intensity representing the accuracy of distinguishing abnormal data based on the abnormal feature values; an identification module, which identifies first abnormal data and suspected abnormal data in the power time-series data based on the parallel detection intensity, and filters out second abnormal data from the suspected abnormal data based on target conditions; and a cleaning and filling module, which deletes abnormal data for the data points corresponding to the first and second abnormal data in the time series, and uses non-abnormal data within a set time window to perform data prediction and filling at the location of the data points to obtain cleaned power time-series data.

[0014] Compared with the prior art, the above-described solution of this application has at least the following beneficial effects: This application provides a time-series detection and cleaning method for power anomaly data. This method generates dynamic prediction indicators by using a time-series correlation analysis model, providing a benchmark for anomaly detection that conforms to the actual operating rules of the power system, thereby effectively identifying potential anomalies that deviate from the trend. Furthermore, by calculating the dual indicators of anomaly feature values ​​and parallel detection intensity, the judgment process simultaneously considers the magnitude of the deviation and its reliability, significantly reducing the risk of false positives. Further, hierarchical identification is performed using detection intensity, first locking in high-confidence anomalies, and then reviewing and screening suspicious data, improving the detection recall rate while controlling the false alarm rate. Finally, based on the normal data before and after the anomaly point, intelligent filling is performed through a prediction model, ensuring that the repaired values ​​closely match the time-series trend of the original data, removing interference while maximizing the integrity and continuity of the data sequence. Attached Figure Description

[0015] Figure 1 A flowchart of a time-series detection and cleaning method for power anomaly data is shown; Figure 2 This diagram illustrates an original time series containing outliers. Figure 3 A schematic diagram of a time series obtained after cleaning is shown; Figure 4 A unit block diagram of a power anomaly data timing detection and cleaning system according to an embodiment of this application is shown. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “said,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0018] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0019] It should be understood that although the terms first, second, third, etc., may be used in the embodiments of this application, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, first may also be referred to as second without departing from the scope of the embodiments of this application, and similarly, second may also be referred to as first.

[0020] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0021] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0022] It should be noted that any symbols and / or numbers present in the specification that are not marked in the accompanying drawings are not reference numerals.

[0023] One approach in this related technology attempts to construct a hierarchical power grid data detection system. This system utilizes multi-level perceptrons to complete the data reception and anomaly identification process, and employs a wolf pack algorithm (WCA) and a genetic algorithm to optimize initial weights, thereby avoiding getting trapped in local optima. This method performs well in static electricity consumption behavior analysis scenarios (such as electricity theft). However, this approach neglects the temporal dependence of power data, making it impossible to model based on its temporal correlations. Furthermore, it requires a large amount of labeled data; its supervised learning requires a large number of labeled anomaly data samples, but power anomaly data is scarce and labeling is costly.

[0024] Another approach in related technologies considers constructing a binary tree (iTree) by randomly partitioning the attribute space. Outliers, due to their significant differences from the cluster, are isolated more quickly; that is, anomalies are determined by calculating the path length from the sample to the root node. This method is suitable for rapid initial screening of large-scale electricity consumption data, enabling coarse detection with higher efficiency. However, this approach shuffles the data order for random partitioning, completely ignoring timestamp information. This leads to the inability to detect collective anomalies formed by consecutive points and a tendency to misjudge periodic peaks as anomalies (such as afternoon load peaks in summer). Furthermore, judging anomalies solely based on numerical distribution fails to distinguish temporal context differences. Therefore, this approach is unsuitable for scenarios requiring contextual analysis, such as frequency fluctuations and transient disturbances.

[0025] In view of this, this application provides a time-series detection and cleaning method for power anomaly data to solve the above-mentioned technical problems.

[0026] The optional embodiments of this application are described in detail below with reference to the accompanying drawings.

[0027] The embodiments provided in this application are embodiments of a method for time-series detection and cleaning of power anomaly data.

[0028] The following is combined with Figure 1 The embodiments of this application will be described in detail.

[0029] Figure 1 A flowchart of a time-series detection and cleaning method for power anomaly data is shown, such as... Figure 1 As shown, it includes the following steps; S101. Using power time-series data as input, determine the current time-series electrical signal prediction index based on the time-series correlation analysis model.

[0030] S102. Based on the electrical signal prediction index and the actual monitoring data at the current moment, determine the abnormal feature values ​​that characterize the difference between the data and the conventional time series trend, and based on the abnormal feature values, determine the parallel detection intensity that characterizes the accuracy of distinguishing abnormal data.

[0031] S103. Based on parallel detection intensity, identify the first abnormal data and suspected abnormal data in the power time series data, and filter out the second abnormal data from the suspected abnormal data based on the target conditions.

[0032] For example, the target condition is set to evaluate the anomalies of data points in a continuous time series. Specifically, for data points identified as suspected anomalies, within a set time window, the trend of non-anomaly data at each time point before and after the data point is compared. If the trend consistency is lower than a preset consistency threshold, the data point is removed; otherwise, it is retained.

[0033] S104. For the data points corresponding to the first and second abnormal data in the time series, delete the abnormal data, use the non-abnormal data within the set time window to which each data point belongs, perform data prediction and fill in the data points to obtain the cleaned power time series data.

[0034] The method provided in this application determines electrical signal prediction indicators based on a time-series correlation analysis model. By learning from historical data, it can capture and model the inherent periodicity and trends of power data. The resulting prediction indicators are dynamic benchmarks that closely reflect actual operating patterns, solving the problem in related technologies where algorithms neglect timestamp information, making it difficult to adapt to complex time-series correlations. Furthermore, through dual judgment and hierarchical identification using anomaly feature values ​​and parallel detection intensity, this method constructs a refined judgment process through secondary screening. This not only reduces the dependence on a large number of pre-labeled anomaly samples but also addresses the problems of high dependence on labeled data and high false positive and false negative rates in existing supervised learning methods by introducing a confidence mechanism and a verification step.

[0035] Furthermore, this method employs a prediction-filling approach based on contextually non-anomaly data for cleaning. This ensures that the cleaned data sequence eliminates the interference of outliers while preserving the temporal characteristics and physical meaning of the original data to the greatest extent possible. This provides a high-quality and coherent data foundation for subsequent advanced applications such as load forecasting and state analysis, overcoming the problems of data sequence breakage caused by simply deleting outliers in related technologies, or the use of simple interpolation such as the mean to mask the true trend.

[0036] In some specific embodiments, in step S101, the time-series correlation analysis model is expressed by the following formula: ; in, for A time-series-based electrical signal prediction index, used to reflect the time-series trend fit value of the data at the current moment. , This represents the coding coefficients of two different power anomaly data. Corresponding to equipment failure type exceptions, Corresponding to communication interference anomalies, , They are respectively Coding coefficients at time 1 , The corresponding data sequence parameters, i.e., the characteristic data of this type of anomaly, , They are coding coefficients , The corresponding power anomaly big data prediction permissions are based on historical anomaly frequency calibration, and the range is... The higher the frequency, the greater the permission weight. , They are coding coefficients , The corresponding time series trend function, where , For trend slope, The intercepts are all obtained by fitting historical non-anomaly data.

[0037] In this embodiment of the application, in steps S101 and S102, power time series data and actual monitoring data are not equivalent concepts.

[0038] Specifically, power time-series data can be understood as a complete, time-continuous data set. Actual monitoring data represents the elements currently to be inspected within this set, i.e., a subset that may contain anomalies. The data sources in this set are selected from a candidate set of anomaly data, as described in the corresponding embodiments below. Furthermore, corresponding to the anomaly-containing data, the non-anomaly data in the power time-series data is defined as actual non-anomaly data. This data is identified as qualified and used as historical elements for establishing inspection standards in the embodiments below.

[0039] In some embodiments, power time-series data are initially separated into a non-abnormal data candidate set and an abnormal data candidate set in the following manner.

[0040] As a specific implementation method, the discretization correction permission is determined based on the time series correlation analysis model, the Hadoop parallel processing platform is built based on the discretization correction permission, and the power time series data is subjected to time-by-time parallel detection based on the Hadoop parallel processing platform to initially separate the power time series data into a non-abnormal data candidate set and an abnormal data candidate set.

[0041] In conjunction with the above embodiments, in step S102, the actual monitoring data at the current moment is selected from the abnormal data candidate set in this embodiment.

[0042] As a feasible implementation, a Hadoop parallel processing platform can be built that includes three types of components: HBase, Hive, and Sqoop.

[0043] For example, to support the above separation method, the HBase component can be used to store structured power data; the Hive component can be used to filter non-abnormal data candidate sets and abnormal data candidate sets through SQL statements; and the Sqoop component can be used to collect data at the current moment in real time, as well as to synchronize and transmit historical databases.

[0044] The structured power data stored in the HBase component includes actual non-abnormal data, defined large-scale power anomalies, and discretized correction permissions.

[0045] In some specific embodiments, SQL statement filtering satisfies Non-abnormal data candidate set and A candidate set of abnormal data.

[0046] in, This indicates the permission to modify the discretization settings.

[0047] In this embodiment, the Hadoop parallel processing platform dynamically adjusts the distribution ratio of normal and abnormal data by discretizing the correction permissions, thereby enabling time-by-time parallel detection of the input mixed data containing non-abnormal and abnormal data, accurately distinguishing the electrical signals at abnormal nodes, and achieving precise separation of non-abnormal and abnormal data.

[0048] In the above embodiments, the discretization correction permission is used to separate anti-interference and anomaly warning.

[0049] In some specific embodiments, the discretization correction authority is determined using the following formula: ; in, Indicates the permission to modify discretization. This represents a predefined, actual, non-abnormal data. This represents the average of the actual non-abnormal data that the power control unit can record per unit of time. This represents the initial value of the discretization distribution coefficient for large-scale power anomaly data, which is calibrated based on the standard deviation of historical anomaly data. Represent a Big data on power anomalies with defined time points. This represents the power data correction ratio based on the time series prediction model, which is determined by the deviation between historical non-abnormal data and the predicted value.

[0050] in, The dimensionality reduction prediction coefficients are calculated using the following formula: ; in, Represents the dimensionality reduction prediction coefficient. and This represents a scaling parameter for two randomly selected large datasets of power anomalies, used to reflect the dispersion of this type of anomaly data, and satisfies... , The minimum dimensionality reduction coefficient, and These are the prediction permission values ​​for actual non-abnormal data and large-scale abnormal power data, respectively, both obtained through historical data statistics. The temporal correlation coefficient is used to reflect the temporal correlation between two types of outlier data, satisfying the following conditions: The stronger the correlation, the larger its value.

[0051] In this application, in step S102, the abnormal feature value and the parallel detection intensity are determined in the following ways.

[0052] In some embodiments, the abnormal feature value is determined using the following formula: .

[0053] in, Indicates abnormal feature values, This represents the coefficient of abnormal behavior in power big data. This indicates the stable fluctuation frequency of abnormal power consumption indicators, calculated as the average number of fluctuations per hour based on historical abnormal data. This represents the unit fluctuation period of the electrical signal, which is consistent with the data acquisition period.

[0054] In some specific embodiments, the abnormal behavior coefficient is set to 1.2 for equipment failure anomalies and 0.8 for communication interference anomalies.

[0055] In some embodiments, the parallel detection intensity is determined using the following formula: .

[0056] in, Indicates the parallel detection intensity. The abnormal characteristics are The power signal transmission vector at time reflects the direction of data change in the time dimension. This represents a predetermined big data labeling coefficient for power anomalies; the industry standard uses values ​​between 0 and 95. , ... express The parallel transmission permissions for different power anomaly big data are allocated by the load distribution of each node on the Hadoop platform, with a total value of 1. This represents the maximum prediction authority for power big data under time series conditions, with a value of 1.0. This represents a given power signal detection scaling parameter, calibrated based on model training error, within a certain range. .

[0057] In some embodiments, step S103, based on parallel detection intensity, identifies first abnormal data and suspected abnormal data in power time-series data, including: dividing the collected historical power data into a training set and a validation set; using the training set to train a time-series correlation analysis model with the training objective of minimizing the mean square error between the power signal prediction index and the actual non-abnormal data; using the trained time-series correlation analysis model to predict the power time-series data time-by-time to obtain a baseline curve of the power signal prediction index changing over time; for each moment of actual monitoring data, if the abnormal feature value corresponding to the actual monitoring data is less than or equal to a first threshold, it is determined to be non-abnormal candidate data; if the abnormal feature value corresponding to the actual monitoring data is greater than the first threshold, and the parallel detection intensity corresponding to the actual monitoring data is greater than or equal to a second threshold calibrated by the validation set, it is determined to be first abnormal data. If the abnormal feature value corresponding to the actual monitoring data is greater than the first threshold, and the parallel detection intensity corresponding to the actual monitoring data is less than the second threshold, it is determined to be suspected abnormal data.

[0058] In some specific embodiments, the collected historical power data is divided into a training set and a validation set in a 7:3 ratio. The training set is used to train the time-series correlation analysis model, and the training objective is to minimize the electrical signal prediction index. Compared with actual non-abnormal data The mean square error is calculated using the following formula: ; in, This represents the mean square error. for Time-series-based electrical signal prediction metrics This represents a defined actual non-abnormal data, and N represents the number of time steps used in the computation.

[0059] For example, when Training stops when the value is less than 0.01 or the number of iterations reaches 1000. Then, the trained model is used to make time-by-time predictions on the full set of time-series data to obtain... The curve that changes over time serves as a benchmark for determining whether the data is abnormal.

[0060] In some specific embodiments, the determination process based on the first threshold can be referenced and implemented based on Table 1, wherein the first threshold is exemplarily set to 0.5.

[0061] Table 1 In some specific embodiments, for the calculated parallel detection intensity Set a second threshold , Calibrate using the validation set data and select the value with the lowest false positive rate on the validation set. value.

[0062] Based on this, the following method is used when making data judgments: like It was determined to be a non-abnormal candidate data; like ,and This indicates that the model's accuracy in distinguishing anomalies in this data meets the standard, and it is officially identified as the first anomaly. like ,and If the data is suspected to be abnormal, its time sequence position will be retained for further verification in subsequent cleaning steps.

[0063] In the above embodiments, the implementation process can be understood as follows: using historical data prior to the period to be determined for training and verification, then generating an ideal curve for the determination period, and then comparing it with the real data for the determination period to detect outliers.

[0064] Among them, non-abnormal candidate data can be incorporated into the non-abnormal data candidate set identified and output by the Hadoop parallel processing platform.

[0065] In some embodiments, step S104, deleting abnormal data and using non-abnormal data within a set time window to which each data point belongs, performs data prediction and filling on the location of the data point, including: deleting data from the location of each data point in the time series; constructing an ARIMA(p, d, q) model based on non-abnormal data within a set time window before and after each data point; performing data prediction on the location of each data point based on the ARIMA(p, d, q) model; calculating the time series trend similarity for each filled data point according to the time window of the data location; and filling the data and completing the cleaning process when the time series trend similarity is greater than or equal to a set similarity threshold.

[0066] In some specific embodiments, for suspected abnormal data, the trend of non-abnormal data is compared with that of three time points before and after it. If the trend consistency is less than 80%, the suspected abnormal data is removed; otherwise, it is retained.

[0067] In this embodiment, non-abnormal data is selected from the non-abnormal candidate set constructed in the above embodiments.

[0068] For example, building an ARIMA(p, d, q) model requires modeling the AR part, I part, and MA part. The following describes the process of building the AR model, I model, MA model, and the overall modeling of the ARIMA(p, d, q) model.

[0069] In some specific embodiments, the AR model is constructed by considering the influence of observations from past periods on the current value, as expressed in the following formula: ; in, Y Represents the observed value. t Indicates the current moment. , , ..., These are model parameters, where c is a constant. It is white noise, and the order p in the formula determines the number of backtracking observations in the model.

[0070] In some specific embodiments, the I-model is constructed by considering the elimination of trends and seasonality in the time series, and its first-difference formula is expressed as follows: ; in, This represents the difference between the observed values ​​at the current time and the previous time.

[0071] In some specific embodiments, the MA model is constructed by considering the impact of past prediction errors on the current value, as expressed in the following formula: ; in, Y Represents the observed value. t Indicates the current moment. , , ..., These are model parameters. It is a constant. It is white noise in the current period. , , ..., It is the white noise from the past, and the order q of the formula determines the amount of white noise in the model's backtracking.

[0072] In this application, based on the modeling of the AR, I, and MA parts of the ARIMA(p, d, q) model described above, the formula for the ARIMA model can be expressed as follows: ; For example, the ARIMA(p, d, q) model can help identify the appropriate order (i.e., hyperparameters p and q) of the autocorrelation function (ACF) and partial autocorrelation function (PACF) plots.

[0073] The order *p* of the autoregressive model is determined by observing the partial autocorrelation function (PACF) plot. Ideally, for an AR(p) model, the PACF plot should exhibit a sudden truncation after lag *p*, meaning the correlation approaches zero after lag *p*. The order *q* of the moving average model is determined by observing the autocorrelation function (ACF) plot. For example, for an MA(q) model, ideally, the ACF plot should show a truncation after lag *q*, implying a significant decrease in correlation after lag *q*. If neither the ACF nor the PACF exhibits a significant tailing or truncation, then the values ​​of *p* and *q* may not be zero. In this case, the optimal *p* and *q* can be selected using AIC or BIC.

[0074] Based on this, the constructed ARIMA(p, d, q) model is obtained through the above steps. Based on the normal time series data before and after the outlier data points, the positions of the proposed outlier points are predicted and filled in. Then, the temporal trend similarity between the repaired data and the original normal data is calculated. The formula is expressed as follows: ; in, This indicates the similarity of the time-series trends between the repaired data and the original normal data, requiring... This is to ensure that the repaired data matches the original trend, thereby achieving a clear understanding of the power anomaly data.

[0075] To facilitate verification of the technical effects achieved by this application, the following is an example of the effect of data detection and cleaning.

[0076] Figure 2 A schematic diagram of an original time series with outliers is shown.

[0077] Figure 3 A schematic diagram of a time series obtained after cleaning is shown.

[0078] For example, this application uses the active power monitoring value of a hydroelectric motor transmitter as an example to repair outlier data errors using time series analysis. The original vibration data is as follows: Figure 2 As shown, outliers were identified, with each outlier corresponding to either a first or second outlier.

[0079] Furthermore, based on the method provided in this application, by deleting and then filling in non-abnormal data based on data trend prediction, each outlier is replaced one by one, so that outliers at each time point in the original dataset are identified and corrected, resulting in... Figure 3 The time series shown. Comparison. Figure 2 and Figure 3 It can be seen that the repaired dataset is more accurate than the original dataset, providing a reliable foundation for subsequent data analysis, equipment performance testing, and fault prediction.

[0080] The method proposed in this application fully considers the impact of past data on current trends, enabling more accurate prediction of signal load curves containing anomalous data. This, in turn, promotes the accurate detection of continuous electrical signals. Simultaneously, it avoids the trend masking issues caused by traditional repair methods and addresses the problem of traditional methods being too rigid and unable to adjust repair strategies according to actual conditions. This makes the repaired dataset more reliable and provides a more comprehensive and efficient data platform for subsequent data analysis.

[0081] This application improves the accuracy of outlier detection by setting dimensionality reduction prediction coefficients, discretization correction permissions, outlier feature values, and parallel detection intensity, making the prediction curve more closely resemble the actual situation. Furthermore, by using the prediction curve derived from the time series model, it achieves a more accurate restoration of the data, enhancing the reference value of the restored data.

[0082] This application also provides system embodiments that follow the above embodiments, for implementing the method steps of the above embodiments. The interpretation of the same names is the same as that of the above embodiments, and they have the same technical effects as those of the above embodiments, so they will not be repeated here.

[0083] like Figure 4 As shown, this application provides a time-series detection and cleaning system 400 for power anomaly data, including: The prediction module 401 is used to determine the electrical signal prediction index at the current moment based on the time series correlation analysis model, using power time series data as input.

[0084] The feature calculation module 402 is used to determine the abnormal feature values ​​that characterize the difference between the data and the conventional time series trend based on the electrical signal prediction index and the actual monitoring data at the current time, and to determine the parallel detection intensity that characterizes the accuracy of distinguishing abnormal data based on the abnormal feature values.

[0085] The identification module 403 is used to identify first abnormal data and suspected abnormal data in power time series data based on parallel detection intensity, and to filter out second abnormal data from the suspected abnormal data based on target conditions.

[0086] The cleaning and filling module 404 is used to delete abnormal data for the data points corresponding to the first abnormal data and the second abnormal data in the time series, and to perform data prediction and filling on the location of the data point using the non-abnormal data within the set time window to which each data point belongs, so as to obtain the cleaned power time series data.

[0087] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0088] Although the operations are described in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the operations shown to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0089] The methods and systems of this application can be implemented using standard programming techniques, utilizing rule-based logic or other logic to implement various method steps. It should also be noted that the terms "system" and "module" as used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.

[0090] Any step, operation, or procedure described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product comprising a computer-readable medium containing computer program code, which is executable by a computer processor to perform any or all of the described steps, operations, or procedures.

[0091] The foregoing description of implementations of this application has been provided for illustrative and descriptive purposes. The foregoing description is not exhaustive and is not intended to limit this application to the exact forms disclosed. Various modifications and variations may exist in accordance with the foregoing teachings, or may arise from practice of this application. These embodiments were chosen and described to illustrate the principles of this application and its practical application, enabling those skilled in the art to utilize this application in various implementations and modifications to suit the specific purpose of the concept.

[0092] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0093] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0094] It is further understood that although the operations are described in a specific order in the accompanying drawings in the embodiments of this application, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all the operations shown to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0095] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the field of this application that are not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0096] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

[0097] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for time-series detection and cleaning of power anomaly data, characterized in that, include: Using power time-series data as input, the electrical signal prediction index for the current moment is determined based on a time-series correlation analysis model; Based on the electrical signal prediction index and the actual monitoring data at the current moment, abnormal feature values ​​that characterize the difference between the data and the conventional time series trend are determined, and based on the abnormal feature values, the parallel detection intensity that characterizes the accuracy of distinguishing abnormal data is determined. Based on the parallel detection intensity, first abnormal data and suspected abnormal data are identified in the power time series data, and second abnormal data are selected from the suspected abnormal data based on the target conditions. For the data points corresponding to the first abnormal data and the second abnormal data in the time series, the abnormal data is deleted, and the non-abnormal data within the set time window to which each data point belongs is used to perform data prediction and fill in the data points to obtain cleaned power time series data.

2. The method according to claim 1, characterized in that, The time-series correlation analysis model is expressed by the following formula: ； in, for Time-series-based electrical signal prediction metrics and This represents the coding coefficients of two different power anomaly data. Corresponding to equipment failure type exceptions, Corresponding to communication interference anomalies, and They are respectively Coding coefficients at time 1 and The corresponding data sequence parameters, and They are coding coefficients and Corresponding big data prediction permissions for power anomalies and They are coding coefficients and The corresponding time-series trend function.

3. The method according to claim 2, characterized in that, The method further includes: The discretization correction authority is determined based on the time series correlation analysis model, and the discretization correction authority is used to separate anti-interference and anomaly warning. A Hadoop parallel processing platform is built based on the aforementioned discretization correction permissions; The power time series data is subjected to time-by-time parallel detection based on the Hadoop parallel processing platform to initially separate the power time series data into a non-abnormal data candidate set and an abnormal data candidate set. The actual monitoring data at the current moment is selected from the abnormal data candidate set.

4. The method according to claim 3, characterized in that, The discretization correction authority is determined using the following formula: ； in, Indicates the permission to modify discretization. This represents a predefined, actual, non-abnormal data. This represents the average of the actual non-abnormal data that the power control unit can record per unit of time. This represents the initial value of the discretization distribution coefficient for large-scale power anomaly data. Represent a Big data on power anomalies with defined time points. This represents the adjustment ratio coefficient for electricity data based on a time series forecasting model. This indicates the calibration characteristics for correcting given electricity data; in, The dimensionality reduction prediction coefficients are calculated using the following formula: ； in, Represents the dimensionality reduction prediction coefficient. and This represents two randomly selected scaling parameters for large datasets of power anomalies. The minimum dimensionality reduction coefficient, and These are the prediction permission values ​​for actual non-abnormal data and large-scale abnormal power data, respectively. For time-series coordination coefficients.

5. The method according to claim 4, characterized in that, The Hadoop parallel processing platform includes: The HBase component is used to store structured power data, which includes the actual non-abnormal data, the defined large data of power anomalies, and the discretized correction permissions. The Hive component is used to filter the non-abnormal data candidate set and the abnormal data candidate set using SQL statements; The Sqoop component is used to collect data in real time and synchronize historical database data.

6. The method according to claim 1, characterized in that, The abnormal feature value is determined using the following formula: ； in, Indicates abnormal feature values, This represents the coefficient of abnormal behavior in power big data. This indicates the stable fluctuation frequency of abnormal power consumption indicators. This represents the unit fluctuation period of the electrical signal.

7. The method according to claim 6, characterized in that, The parallel detection intensity is determined using the following formula: ； in, Indicates the parallel detection intensity. Indicates abnormal feature values The power signal transmission vector, This represents the given big data labeling coefficients for power anomalies. , ... express Different permissions for parallel transmission of big data on power anomalies This indicates the maximum prediction authority for power big data under time series conditions. This represents a predetermined power signal detection scaling parameter.

8. The method according to claim 1, characterized in that, The process of identifying first abnormal data and suspected abnormal data in the power time-series data based on the parallel detection intensity includes: The collected historical power data is divided into a training set and a validation set. The training objective is to minimize the mean square error between the predicted power signal index and the actual non-abnormal data. The training set is used to train the time series correlation analysis model. The trained time-series correlation analysis model is used to predict the power time-series data time by time, and the baseline curve of the power signal prediction index changes over time is obtained. For each moment of actual monitoring data, if the abnormal feature value corresponding to the actual monitoring data is less than or equal to the first threshold, it is determined to be non-abnormal candidate data. If the abnormal feature value corresponding to the actual monitoring data is greater than the first threshold, and the parallel detection intensity corresponding to the actual monitoring data is greater than or equal to the second threshold calibrated by the validation set, then it is determined to be the first abnormal data. If the abnormal feature value corresponding to the actual monitoring data is greater than the first threshold, and if the parallel detection intensity corresponding to the actual monitoring data is less than the second threshold, then it is determined to be the suspected abnormal data.

9. The method according to claim 1, characterized in that, Using non-abnormal data within a defined time window to which each data point belongs, perform data prediction and filling on the location of the data point, including: Remove the data at the locations of each of the aforementioned data points from the time series; Based on the non-abnormal data within a set time window before and after each data point, an ARIMA(p, d, q) model is constructed. Based on the ARIMA(p, d, q) model, data prediction is performed on the location of each data point. For each of the filled data, a time-series trend similarity calculation is performed according to the time window of the data's location; If the time-series trend similarity is greater than or equal to the set similarity threshold, the data will be filled in and the cleaning process will be completed.

10. A time-series detection and cleaning system for power anomaly data, characterized in that, include: The prediction module is used to determine the electrical signal prediction index at the current moment based on the time series correlation analysis model, using power time series data as input. The feature calculation module is used to determine abnormal feature values ​​that characterize the difference between the data and the conventional time series trend based on the electrical signal prediction index and the actual monitoring data at the current time, and to determine the parallel detection intensity that characterizes the accuracy of distinguishing abnormal data based on the abnormal feature values. The identification module is used to identify first abnormal data and suspected abnormal data in the power time series data based on the parallel detection intensity, and to filter out second abnormal data from the suspected abnormal data based on target conditions. The cleaning and filling module is used to delete abnormal data for the data points corresponding to the first abnormal data and the second abnormal data in the time series, and to perform data prediction and filling on the location of the data points using the non-abnormal data within the set time window to which each data point belongs, so as to obtain the cleaned power time series data.

Images