Data exception monitoring method and device, equipment and storage medium

Abstract

The invention discloses a data exception monitoring method and device, equipment and a storage medium. The method comprises the steps of obtaining to-be-monitored data; inputting the to-be-monitored data into a preset multivariate Gaussian distribution model to enable the multivariate Gaussian distribution model to calculate a probability density function value of the to-be-monitored data according to a preset mean value and a covariance matrix, and outputting the probability density function value of the to-be-monitored data; the probability density function value is compared with a preset abnormal threshold value, when the probability density function value is smaller than the abnormal threshold value, it is judged that the to-be-monitored data is abnormal data, and otherwise, it is judged that the to-be-monitored data is normal data; and when it is determined that the to-be-monitored data is abnormal data, generating corresponding data abnormity early warning information. According to the invention, abnormity monitoring and early warning of mass data of the power grid can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of data anomaly monitoring, and in particular to a data anomaly monitoring method, device, equipment and storage medium. Background Art

[0002] With the continuous development and popularization of modern science and technology, data collection has become an indispensable part of various fields. In the process of collecting massive high-dimensional and high-density data, anomaly detection and early warning are particularly important. These data usually contain a lot of information and hidden anomalies, which may have a significant impact on scientific research, industrial production, financial investment and other fields. Especially in the field of power grids, since the equipment in power grid production is not completely reliable and the manually recorded data is prone to errors, the power load data will generate a large amount of abnormal and incomplete data. Due to the large amount of data, the incompleteness and inaccuracy of the data are easily magnified, which in turn affects the subsequent data mining and analysis. Therefore, how to collect, detect anomalies and warn of massive high-dimensional and high-density data in the power grid in real time has become an urgent problem to be solved. Summary of the invention

[0003] The present invention provides a data anomaly monitoring method, device, equipment and storage medium to solve the technical problem that power load data will generate a large amount of abnormal data and incomplete data. Due to the large amount of data, the incompleteness and inaccuracy of the data are easily magnified, thereby affecting subsequent data mining and analysis.

[0004] In order to solve the above technical problems, an embodiment of the present invention provides a data anomaly monitoring method, including:

[0005] Obtain the data to be monitored;

[0006] Inputting the data to be monitored into a preset multivariate Gaussian distribution model, so that the multivariate Gaussian distribution model calculates the probability density function value of the data to be monitored according to a preset mean and covariance matrix, and outputs the probability density function value of the data to be monitored;

[0007] Comparing the probability density function value with a preset abnormal threshold, when the probability density function value is less than the abnormal threshold, determining the data to be monitored to be abnormal data, otherwise determining the data to be monitored to be normal data;

[0008] When the data to be monitored is determined to be abnormal data, corresponding data abnormality warning information is generated;

[0009] The multivariate Gaussian distribution model is obtained by fitting a preset multivariate Gaussian distribution based on historical data.

[0010] As a preferred solution, before inputting the data to be monitored into a preset multivariate Gaussian distribution model, the method further includes:

[0011] The data to be monitored is preprocessed; wherein the preprocessing includes: data cleaning and data standardization.

[0012] As a preferred solution, the generation of the multivariate Gaussian distribution model includes:

[0013] Acquire historical data and preprocess the historical data;

[0014] Extract data features corresponding to the preprocessed historical data;

[0015] Obtaining a preset neighborhood of a multivariate Gaussian distribution and a minimum number of points required to form a high-density area, and then, according to the neighborhood and the minimum number of points, adding, among all the historical data points corresponding to the historical data, the historical data points whose number of remaining historical data points existing within the neighborhood is greater than or equal to the minimum number of points to a preset set, to obtain a corresponding cluster set;

[0016] The mean and covariance matrix of the data features corresponding to the historical data points in the cluster set are calculated, and then a preset multivariate Gaussian distribution is fitted according to the mean and covariance matrix to obtain the multivariate Gaussian distribution model.

[0017] As a preferred solution, according to the neighborhood size and the minimum number of points, among all the historical data points corresponding to the historical data, the historical data points whose number of remaining historical data points existing in the neighborhood is greater than or equal to the minimum number of points are added to a preset set to obtain a corresponding cluster set, including:

[0018] Initializing an empty cluster set and an empty noise point set, and marking all historical data points corresponding to the historical data as unvisited;

[0019] Randomly select a first historical data point marked as unvisited from the historical data points, calculate the number of the remaining historical data points within the neighborhood of the first historical data point, and if the number is less than the minimum number of points, mark the first historical data point as visited, add the first historical data point to the noise point set, and then continue to visit the next historical data point;

[0020] If the number is greater than or equal to the minimum number of points, marking the first historical data point as visited, and adding the first historical data point to the cluster set;

[0021] Recursively access the remaining historical data points within the neighborhood of the first historical data point, mark the accessed historical data points as visited, and calculate the number of remaining historical data points within the neighborhood of the accessed historical data point. If the number is less than the minimum number of points, add the accessed historical data point to the cluster set, otherwise add it to the noise point set, to obtain the corresponding cluster set and noise point set.

[0022] As a preferred solution, the preprocessing of the historical data includes:

[0023] Determine whether there is historical abnormal data in the historical data, and if there is historical abnormal data in the historical data, extract the historical abnormal data;

[0024] According to a preset approximate symbol algorithm, the historical abnormal data is subjected to time series discrete processing and symbolization processing to obtain a symbol sequence corresponding to the historical abnormal data;

[0025] Calculating the similarity between the symbol sequences according to a preset Euclidean distance algorithm, and then obtaining a data reference curve according to the similarity between the symbol sequences;

[0026] According to the data reference curve, the time series corresponding to the historical abnormal data is adjusted.

[0027] As a preferred solution, the determining whether there is any historical abnormal data in the historical data includes:

[0028] Calculate the median of the historical data;

[0029] Calculate the lower quartile and the upper quartile of the historical data, and then calculate the interquartile range of the historical data according to the lower quartile and the upper quartile;

[0030] Calculate the lower outlier threshold and the upper outlier threshold corresponding to the historical data according to the median, the lower quartile, the upper quartile and the interquartile range;

[0031] The historical data is compared with the lower abnormal value threshold and the upper abnormal value threshold, and the historical data that is less than the lower threshold or greater than the upper threshold is taken as the historical abnormal data.

[0032] As a preferred solution, the probability density function value of the data to be monitored is calculated according to the following formula:

[0033]

[0034] Among them, n is the number of data features of the data to be monitored; x is the feature vector containing the currently detected power data parameters; μ is the vector of the feature mean; Σ is the covariance matrix; |Σ| is the determinant of the covariance matrix.

[0035] Based on the above embodiment, another embodiment of the present invention provides a data anomaly monitoring device, including: a to-be-monitored data acquisition module, a probability density function value calculation module, an abnormal data identification module, and a data anomaly warning generation module;

[0036] The module for acquiring data to be monitored is used to acquire the data to be monitored;

[0037] The probability density function value calculation module is used to input the data to be monitored into a preset multivariate Gaussian distribution model, so that the multivariate Gaussian distribution model calculates the probability density function value of the data to be monitored according to the preset mean and covariance matrix, and outputs the probability density function value of the data to be monitored; wherein the multivariate Gaussian distribution model is obtained by fitting a preset multivariate Gaussian distribution according to historical data;

[0038] The abnormal data identification module is used to compare the probability density function value with a preset abnormal threshold value, and when the probability density function value is less than the abnormal threshold value, determine that the data to be monitored is abnormal data, otherwise determine that the data to be monitored is normal data;

[0039] The data anomaly warning generation module is used to generate corresponding data anomaly warning information when it is determined that the data to be monitored is abnormal data.

[0040] Based on the above embodiments, another embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, the data anomaly monitoring method described in the above embodiments of the invention is implemented.

[0041] Based on the above embodiments, another embodiment of the present invention provides a storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the data anomaly monitoring method described in the above invention embodiment.

[0042] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0043] The present invention provides a data anomaly monitoring method, which includes acquiring data to be monitored; inputting the data to be monitored into a preset multivariate Gaussian distribution model, so that the multivariate Gaussian distribution model calculates the probability density function value of the data to be monitored according to a preset mean and covariance matrix, and outputs the probability density function value of the data to be monitored; comparing the probability density function value with a preset anomaly threshold, and when the probability density function value is less than the anomaly threshold, determining that the data to be monitored is abnormal data, otherwise determining that the data to be monitored is normal data; when the data to be monitored is determined to be abnormal data, generating corresponding data anomaly warning information; wherein the multivariate Gaussian distribution model is obtained by fitting a preset multivariate Gaussian distribution according to historical data.

[0044] Through the present invention, abnormal monitoring and early warning of massive data in power grids can be realized. The present invention combines multivariate Gaussian distribution to realize abnormal monitoring of massive data in power grids. In data anomaly detection, multivariate Gaussian distribution has significant advantages. Since data in the real world usually have certain noise and errors, multivariate Gaussian distribution can describe the basic statistical characteristics of the data to be monitored by modeling the mean and covariance of the data to be monitored. If the probability density function value of the data point to be monitored in the distribution is low, that is, less than the preset abnormal threshold, the point can be considered as an abnormal point. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of a data anomaly monitoring method provided by an embodiment of the present invention;

[0046] Figure 2 It is a data cleaning and fusion flow chart of historical data;

[0047] Figure 3 It is a reference table of PAA coefficients;

[0048] Figure 4 It is a structural schematic diagram of a data anomaly monitoring device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0051] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0052] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0053] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0054] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0055] In the description of the embodiments of the present application, unless otherwise clearly specified and limited, technical terms such as "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0056] Embodiment 1

[0057] Please refer to Figure 1, is a flow chart of a data anomaly monitoring method provided by an embodiment of the present invention, comprising the following specific steps:

[0058] S1. Obtain the data to be monitored;

[0059] Preferably, before inputting the data to be monitored into a preset multivariate Gaussian distribution model, the method further includes: preprocessing the data to be monitored; wherein the preprocessing includes: data cleaning and data standardization.

[0060] Specifically, the data to be monitored include various distribution network power measurement parameters, such as voltage, current, power, power factor, and electric energy. There are many ways to obtain this data, including:

[0061] (1) SCADA system acquisition: Supervisory Control And Data Acquisition (SCADA) is a system that monitors and controls the distribution network in real time. It can collect data on various power parameters and perform processing and analysis.

[0062] (2) Smart meter data: Smart meters can measure and record each user's electricity consumption data, including electricity consumption, power, and voltage. These data can be transmitted to the data platform via the network.

[0063] Among them, the data type types of distribution network data can be divided into two categories: actuator data and sensor data. Taking the energy storage system (ESS) in the power system as an example, the actuators include bidirectional power electronic transformers and circuit breakers, and the sensors include voltage, current and temperature sensors. However, the analysis of a large amount of measurement data is a bottleneck for the time performance of anomaly detection. For example, a PMU generates up to 60 timestamp samples per second, and the data accumulated by the PMU is in the range of several megabits per day. Such a huge amount of data poses a challenge to the automation of anomaly detection in large-scale smart grids. On the other hand, for critical infrastructure such as power grids and communication networks, sensors generate a large amount of complex time series data, and often have high-dimensional features. As sensor data becomes more complex and the dimensionality increases, it is difficult to meet the requirements for processing efficiency and effect using traditional monitoring and anomaly capture methods.

[0064] In the context of power grids, there are many reasons for the generation of abnormal values. In the entire power system, abnormalities may occur in the power generation, transmission, energy storage and power consumption stages, such as equipment failure in the energy storage system, network attacks on the dispatching control center, and power theft on the power consumption side. These abnormal data often have significant differences in distribution and timing patterns from adjacent normal data, and are generally collected by the above-mentioned SCADA systems and PMUs and other devices for further analysis.

[0065] The present invention mainly targets power grid time series data, which usually comes from various sensors, meters and measuring equipment, including power load, voltage, current, power and electric energy data. Its power physical parameters include voltage (V), current (I), power (P), frequency (f), etc.:

[0066] P: Power, in watts (W);

[0067] V: Voltage (Voltage), the unit is volt (V);

[0068] I: Current, in ampere (A);

[0069] f: frequency (Frequency), the unit is Hertz (Hz);

[0070] Q: Reactive Power, in vars;

[0071] S: Apparent Power, in volt-amperes (VA);

[0072] Specifically, when the data anomaly monitoring method of the present invention is used to perform anomaly monitoring and early warning on the power grid data to be monitored, the specific steps include:

[0073] (1) First, collect the various parameter data of the power system during operation, including power, voltage, current, frequency, reactive power and apparent power. Then perform necessary data preprocessing, such as data cleaning to remove invalid or erroneous data points, and data standardization.

[0074] S2. Input the data to be monitored into a preset multivariate Gaussian distribution model, so that the multivariate Gaussian distribution model calculates the probability density function value of the data to be monitored according to the preset mean and covariance matrix, and outputs the probability density function value of the data to be monitored; wherein the multivariate Gaussian distribution model is obtained by fitting a preset multivariate Gaussian distribution according to historical data;

[0075] Preferably, the probability density function value of the data to be monitored is calculated according to the following formula:

[0076]

[0077] Among them, n is the number of data features of the data to be monitored; x is the feature vector containing the currently detected power data parameters; μ is the vector of the feature mean; Σ is the covariance matrix; |Σ| is the determinant of the covariance matrix.

[0078] (2) For the data point x to be monitored, input it into the preset multivariate Gaussian distribution model and calculate its probability density function value p(x) under the multivariate Gaussian distribution. The calculation formula is as follows:

[0079]

[0080] Wherein, n is the number of features; x is the feature vector containing the currently detected power data parameters; μ is the vector of feature means; Σ is the covariance matrix; |Σ| is the determinant of the covariance matrix.

[0081] Preferably, the generation of the multivariate Gaussian distribution model includes: obtaining historical data and preprocessing the historical data; extracting data features corresponding to the preprocessed historical data; obtaining a preset neighborhood of the multivariate Gaussian distribution and the minimum number of points required to form a high-density area, and then, based on the neighborhood and the minimum number of points, adding the historical data points corresponding to the historical data and the number of remaining historical data points existing in the neighborhood range greater than or equal to the minimum number of points to a preset set to obtain a corresponding cluster set; calculating the mean and covariance matrix of the data features corresponding to the historical data points in the cluster set, and then fitting a preset multivariate Gaussian distribution based on the mean and covariance matrix to obtain the multivariate Gaussian distribution model.

[0082] Preferably, according to the neighborhood size and the minimum number of points, among all the historical data points corresponding to the historical data, the historical data points whose number of remaining historical data points existing in the neighborhood is greater than or equal to the minimum number of points are added to a preset set to obtain a corresponding cluster set, including: initializing an empty cluster set and an empty noise point set, and marking all the historical data points corresponding to the historical data as unvisited; randomly selecting a first historical data point marked as unvisited from the historical data points, calculating the number of remaining historical data points of the first historical data point in the neighborhood, and if the number is less than the minimum number of points, marking the first historical data point as visited. The process of recursively accessing the remaining historical data points within the neighborhood of the first historical data point, marking the accessed historical data points as accessed, and calculating the number of remaining historical data points within the neighborhood of the accessed historical data point. If the number is less than the minimum number of points, adding the accessed historical data point to the cluster set, otherwise adding it to the noise point set, to obtain the corresponding cluster set and noise point set.

[0083] Specifically, the fitting steps of the multivariate Gaussian distribution model are as follows:

[0084] (1) Data collection and preprocessing: Collect historical data of the power system, including various parameter data of the power system during operation, including power, voltage, current, frequency, reactive power and apparent power. Perform necessary data preprocessing, such as data cleaning and data fusion.

[0085] (2) Feature extraction: extracting computational features from the historical data, for example:

[0086] Pmax, Pmin: maximum and minimum power values;

[0087] Vmax, Vmin: maximum and minimum voltage values;

[0088] Imax, Imin: maximum and minimum current values;

[0089] fmax, fmin: maximum and minimum frequency;

[0090] Qmax, Qmin: maximum and minimum values ​​of reactive power;

[0091] Smax, Smin: maximum and minimum values ​​of apparent power;

[0092] (3) Divide the cluster set and the noise point set:

[0093] 1. Parameter definition

[0094] ε(eps): neighborhood size. In a multivariate Gaussian distribution, this can be interpreted as the threshold of the deviation of a data point from the mean.

[0095] MinPts: The minimum number of points required to form a high-density area, corresponding to the threshold of the probability density function in the multivariate Gaussian distribution.

[0096] 2. Initialization

[0097] Mark all power data points as "not visited".

[0098] Initialize an empty set of clusters and an empty set of noise points.

[0099] 3. Select the starting point

[0100] Randomly select an “unvisited” data point p from the electricity dataset.

[0101] 4. Neighborhood of the access point

[0102] Retrieve all power data points in the ε-neighborhood of point p, that is, points whose probability density function value difference with p in the multivariate Gaussian distribution is less than or equal to ε.

[0103] 5. Expand the cluster

[0104] If the number of data points in the ε-neighborhood of point p is greater than or equal to MinPts, p is considered to belong to a high-density area, marked as visited, and a new cluster C is created and p is added to cluster C.

[0105] For each point p' in p's ε-neighborhood, if p' is "unvisited", it is marked as visited and a decision is made whether to add it to cluster C based on its similarity to p (the probability density function value in the multivariate Gaussian distribution).

[0106] 6. Process the remaining points

[0107] Repeat steps 3 to 5 until all power data points have been visited.

[0108] 7. Identify noise points

[0109] After all power data points are visited and assigned to clusters or marked as noise points, those points that neither belong to any cluster nor are marked as “visited” are noise points.

[0110] 8. Output

[0111] Output the set of all clusters and the set of noise points.

[0112] Through the above steps, it is possible to identify noise points in the historical data that do not belong to any cluster, which indicate equipment failures, operational errors, or data errors. The output cluster set represents high-density areas in the data, which usually correspond to the normal operating mode of the power system. Each cluster can be further analyzed to determine the characteristic mean and covariance under normal operating conditions, which helps to understand the behavior of the system in a stable state.

[0113] (4) Calculate the mean and covariance matrix: Use the historical data points in the cluster set to calculate the mean μ and covariance matrix Σ of the feature;

[0114] The mean μ describes the average value of each power data parameter under normal conditions, while the covariance matrix Σ describes the correlation between these parameters. Using the calculated mean and covariance matrix, new power data points can be scored to determine whether they conform to the multivariate Gaussian distribution of the normal operating mode. Calculate the mean μ and covariance matrix Σ of the extracted features: Each element of the mean vector μ is the average value of the corresponding feature in the historical normal data set. The covariance matrix Σ describes the correlation between features.

[0115] Then, according to the mean and covariance matrix, a preset multivariate Gaussian distribution is fitted to obtain the multivariate Gaussian distribution model. When using the multivariate Gaussian distribution model for data anomaly monitoring and early warning, for the data point x to be monitored, its probability density function value p(x) under the multivariate Gaussian distribution is calculated.

[0116] (5) Threshold judgment: Set an abnormal threshold ∈. If p(x) < ∈, the data point x is considered abnormal; otherwise, it is considered normal. This allows anomaly detection of real-time data collected at the minute level without the need for abnormal data annotation. It supports online anomaly detection of high-dimensional and high-density data collected in real time and directly feeds back to the main station for early warning.

[0117] Preferably, the preprocessing of the historical data includes: determining whether there is historical abnormal data in the historical data, and extracting the historical abnormal data when there is historical abnormal data in the historical data; performing time series discretization processing and symbolization processing on the historical abnormal data according to a preset approximate symbol algorithm to obtain a symbol sequence corresponding to the historical abnormal data; calculating the similarity between each of the symbol sequences according to a preset Euclidean distance algorithm, and then obtaining a data reference curve according to the similarity between each of the symbol sequences; and adjusting the time series corresponding to the historical abnormal data according to the data reference curve.

[0118] Preferably, the determining whether there is historical abnormal data in the historical data includes: calculating the median of the historical data; calculating the lower quartile and the upper quartile of the historical data, and then calculating the interquartile range of the historical data based on the lower quartile and the upper quartile; calculating the lower outlier threshold and the upper outlier threshold corresponding to the historical data based on the median, the lower quartile, the upper quartile and the interquartile range; comparing the historical data with the lower outlier threshold and the upper outlier threshold, and treating the historical data that is less than the lower threshold or greater than the upper threshold as the historical abnormal data.

[0119] Specifically, after collecting the historical data of the power grid, it is randomly divided into a training set and a test set, and then a multivariate Gaussian distribution model is fitted under the training set. The historical data is normal operation data, and the multivariate Gaussian distribution model is fitted using normal operation, and data anomaly monitoring can be achieved without abnormal data annotation. However, since the equipment in power grid production is not completely reliable, data errors may also exist in the historical data. According to the characteristics of power big data, data errors can be divided into the following categories.

[0120] (1) Duplicate data: Duplicate data can be divided into two categories: duplicate data attributes and duplicate data instances. Due to the fluctuation of power grid status, duplicate data attribute problems mainly come from business definition or code duplication, and duplicate data instances mainly come from duplicate records generated by the detection system.

[0121] (2) Abnormal data: Abnormal data is divided into invalid data and unqualified data. Invalid data mainly refers to data where all or most instances are null or default values, mainly from reserved fields or tag fields of certain data tables in the power grid system. Unqualified data mainly refers to instances that violate the rules in the data or instances that exceed the specified set range, which is mainly caused by data transmission or manual errors.

[0122] (3) Missing data: The main reasons for missing data are fluctuations in the entry of power grid data into the system, data transmission loss, or manual omissions. From the perspective of the entire data, the number of missing instances is relatively small.

[0123] Therefore, before fitting the multivariate Gaussian distribution model based on the historical data, the historical data needs to be cleaned to ensure the integrity and accuracy of the data.

[0124] Please refer to Figure 2 , which is a flowchart of data cleaning and fusion of historical data, is divided into 4 steps: first, the approximate symbol aggregation algorithm is used to discretize and symbolize the time series of 1440 data points; secondly, the Euclidean distance algorithm is used to calculate the similarity between symbol sequences; then, according to the curve fitted by the similar sequence, the 1440 data are used to complete the identification, correction and filling of abnormal data and missing data; finally, the cleaned data is fused. The specific steps are as follows:

[0125] (1) Determine whether there are outliers or missing values ​​in the data:

[0126] Assuming that we already have the numerical representation of the symbolic sequence of historical data (i.e., the original numerical value before the SAX algorithm), we can calculate the median (Med) and interquartile range (IQR = Q3-Q1, where Q3 is the upper quartile and Q1 is the lower quartile) of each time point:

[0127] Step 1: Calculate the median Med of the historical data set;

[0128] Step 2, calculate the lower quartile Q1 and the upper quartile Q3;

[0129] Step 3, calculate the interquartile range IQR = Q3-Q1;

[0130] Step 4: Calculate the lower outlier threshold and the upper outlier threshold according to the selected k value;

[0131] The calculation formula for the lower outlier threshold (LowerThreshold) is:

[0132] textLowerThreshold=Med-k×IQR;

[0133] The calculation formula for the upper outlier threshold (UpperThreshold) is:

[0134] textUpperThreshold=Med+k×IQR;

[0135] Where Med is the median, IQR is the interquartile range, and k is a constant (usually 1.5 or 3). Q1 is the lower quartile, and Q3 is the upper quartile. The interquartile range IQR is defined as:

[0136] IQR = Q3-Q1;

[0137] In practical applications, Q1 and Q3 represent the values ​​at the 25% and 75% positions of all values ​​in the data set after all values ​​are arranged from small to large, respectively. The median Med is the value at the 50% position.

[0138] Step 5: Compare the values ​​at each time point with the two thresholds. Data points below the lower threshold or above the upper threshold can be considered as outliers.

[0139] (2) Use approximate symbolic algorithms to discretize and symbolize the time series data:

[0140] The SAX algorithm is used to discretize and symbolize the time series data. First, the original time series data is normalized so that its mean is 0. Then, the time series is converted into a representation with a lower dimension using piecewise aggregate approximation (PAA). Finally, the PAA coefficients are converted into letters through a table lookup method to obtain a symbolic representation of the time series.

[0141] The following are the steps to use the SAX algorithm to discretize and symbolize time series data:

[0142] Step 1: Normalize the original time series data so that its mean is 0. The specific formula is as follows:

[0143]

[0144] Among them, x i is the ith data point in the original time series, is the mean of the original time series, and s is the standard deviation of the original time series.

[0145] Step 2: Use piecewise aggregate approximation (PAA) to transform the time series into a representation with lower dimensions. The specific formula is as follows:

[0146]

[0147] Among them, X j is the j-th data point of the transformed time series, w is the window width, and x i is the i-th data point in the original time series.

[0148] Step 3: Convert the PAA coefficients into letters by means of a look-up table method to obtain the symbolic representation of the time series. The specific formula is as follows:

[0149] y j = f(X j );

[0150] Among them, y j is the j-th data point of the symbolized time series, and f is a mapping function that maps the PAA coefficients to letters. This Step 3 involves a look-up table method, that is, mapping the PAA coefficients to letters to achieve the symbolic representation.

[0151] Mapping the PAA coefficients to letters can achieve the symbolic representation by means of a look-up table method. The specific look-up table is as Figure 3 shown. Among them, the range of the PAA coefficients is calculated based on the original time series data, and each range corresponds to a letter. By mapping the PAA coefficients to letters, the high-dimensional time series data can be converted into a low-dimensional symbolic representation.

[0152] (3) Calculate the similarity between symbol sequences using the Euclidean distance algorithm:

[0153] For the corresponding symbols in two symbol sequences, use a predefined letter order (for example, A < B < C < D < E) to convert each symbol into its position in the alphabet (for example, A = 1, B = 2,..., E = 5). Then, regard these positions as numerical values and calculate the Euclidean distance between the two sequences.

[0154] The specific Euclidean distance calculation formula is:

[0155]

[0156] Among them, S1 and S2 are two symbol sequences, and s 1i and s 2i are the numerical positions of the i-th symbol in sequences S1 and S2 respectively, and n is the length of the symbol sequence.

[0157] The smaller the Euclidean distance, the more similar the two symbol sequences are. Therefore, the similarity between the two sequences can be evaluated by calculating the distance between them.

[0158] In the application process, the length of the 1440-point daily load power data is fixed (the sampling period is set to 1 minute, and the daily load sequence is 1440 points). SAX is used to convert the 1440-point data into characters. The Euclidean distance can quickly calculate the similarity of the SAX symbolic expression at a low computational complexity. The greater the distance between the SAX expressions of two 1440-point sequences, the lower their similarity. Therefore, the similarity of the two 1440-point time series curves is:

[0159]

[0160] Where: Q and C are two time series respectively; qi is the i-th point of the Q series, and cj is the j-th point of the C series.

[0161] (4) Adjust the similar curve and replace it with the reference curve value:

[0162] After performing symbolic approximate aggregation and similarity measurement on the time series, we obtain | ω similar time series A and similar time series SXA to be cleaned, where X is the original sequence, | ω Usually, the number of time series in a month is 30. The similar time series A is adjusted by the weighted adjustment method (FittedCurve algorithm) to obtain the reference curve relative to the original time series X If there are missing values ​​in the original time series, they are filled with the values ​​of the corresponding points in the reference curve, and then the filled time series is fused with multivariate heterogeneous data to obtain the processed historical data.

[0163] S3, comparing the probability density function value with a preset abnormal threshold, when the probability density function value is less than the abnormal threshold, determining that the data to be monitored is abnormal data, otherwise determining that the data to be monitored is normal data;

[0164] (3) Then, according to the set abnormal threshold ∈, the probability density function value p(x) is compared with the abnormal threshold. If p(x)<∈, the data point x is considered abnormal; otherwise, it is considered normal, thereby realizing anomaly detection of real-time collected data at the minute level without the need for abnormal data labeling.

[0165] S4. When it is determined that the data to be monitored is abnormal data, corresponding data abnormality warning information is generated.

[0166] (4) When the data to be monitored is determined to be abnormal data, that is, when abnormal data exists, corresponding data abnormality warning information is generated and directly fed back to the main station for warning. Among them, the warning method can be customized according to actual needs, including SMS, email, system prompts and other methods. By feeding back abnormal data in real time, risks can be discovered in time and handled accordingly, thereby improving operational efficiency and risk management level.

[0167] Embodiment 2

[0168] Please refer to Figure 4 , is a schematic diagram of the structure of a data anomaly monitoring device provided by an embodiment of the present invention, the device comprising: a to-be-monitored data acquisition module, a probability density function value calculation module, an abnormal data identification module and a data anomaly warning generation module;

[0169] The module for acquiring data to be monitored is used to acquire the data to be monitored;

[0170] The probability density function value calculation module is used to input the data to be monitored into a preset multivariate Gaussian distribution model, so that the multivariate Gaussian distribution model calculates the probability density function value of the data to be monitored according to the preset mean and covariance matrix, and outputs the probability density function value of the data to be monitored; wherein the multivariate Gaussian distribution model is obtained by fitting a preset multivariate Gaussian distribution according to historical data;

[0171] The abnormal data identification module is used to compare the probability density function value with a preset abnormal threshold value, and when the probability density function value is less than the abnormal threshold value, determine that the data to be monitored is abnormal data, otherwise determine that the data to be monitored is normal data;

[0172] The data anomaly warning generation module is used to generate corresponding data anomaly warning information when it is determined that the data to be monitored is abnormal data.

[0173] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0174] Those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0175] Embodiment 3

[0176] Accordingly, an embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the data anomaly monitoring method described in the above-mentioned embodiment of the invention when executing the computer program.

[0177] The electronic device may be a computing device such as a desktop computer, a notebook, a palm computer, a cloud server, etc. The device may include, but is not limited to, a processor and a memory.

[0178] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the device, and various interfaces and lines are used to connect various parts of the entire device.

[0179] Embodiment 4

[0180] Accordingly, an embodiment of the present invention provides a storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the data anomaly monitoring method described in the above-mentioned embodiment of the invention.

[0181] The memory can be used to store the computer program, and the processor realizes various functions of the device by running or executing the computer program stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0182] The storage medium is a computer-readable storage medium, and the computer program is stored in the computer-readable storage medium. When the computer program is executed by the processor, the steps of each method embodiment described above can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0183] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A data anomaly monitoring method, characterized in that: include: Obtain the data to be monitored; Inputting the data to be monitored into a preset multivariate Gaussian distribution model, so that the multivariate Gaussian distribution model calculates the probability density function value of the data to be monitored according to a preset mean and covariance matrix, and outputs the probability density function value of the data to be monitored; Comparing the probability density function value with a preset abnormal threshold, when the probability density function value is less than the abnormal threshold, determining the data to be monitored to be abnormal data, otherwise determining the data to be monitored to be normal data; When the data to be monitored is determined to be abnormal data, corresponding data abnormality warning information is generated; The multivariate Gaussian distribution model is obtained by fitting a preset multivariate Gaussian distribution based on historical data.

2. The data anomaly monitoring method according to claim 1, characterized in that: Before inputting the data to be monitored into a preset multivariate Gaussian distribution model, the method further includes: The data to be monitored is preprocessed; wherein the preprocessing includes: data cleaning and data standardization.

3. The data anomaly monitoring method according to claim 1, characterized in that: The generation of the multivariate Gaussian distribution model includes: Acquire historical data and preprocess the historical data; Extract data features corresponding to the preprocessed historical data; Obtaining a preset neighborhood of a multivariate Gaussian distribution and a minimum number of points required to form a high-density area, and then, according to the neighborhood and the minimum number of points, adding, among all the historical data points corresponding to the historical data, the historical data points whose number of remaining historical data points existing within the neighborhood is greater than or equal to the minimum number of points to a preset set, to obtain a corresponding cluster set; The mean and covariance matrix of the data features corresponding to the historical data points in the cluster set are calculated, and then a preset multivariate Gaussian distribution is fitted according to the mean and covariance matrix to obtain the multivariate Gaussian distribution model.

4. The data anomaly monitoring method according to claim 3, characterized in that: According to the neighborhood size and the minimum number of points, among all the historical data points corresponding to the historical data, the historical data points whose number of other historical data points existing in the neighborhood is greater than or equal to the minimum number of points are added to a preset set to obtain a corresponding cluster set, including: Initializing an empty cluster set and an empty noise point set, and marking all historical data points corresponding to the historical data as unvisited; Randomly select a first historical data point marked as unvisited from the historical data points, calculate the number of the remaining historical data points within the neighborhood of the first historical data point, and if the number is less than the minimum number of points, mark the first historical data point as visited, add the first historical data point to the noise point set, and then continue to visit the next historical data point; If the number is greater than or equal to the minimum number of points, marking the first historical data point as visited, and adding the first historical data point to the cluster set; Recursively access the remaining historical data points within the neighborhood of the first historical data point, mark the accessed historical data points as visited, and calculate the number of remaining historical data points within the neighborhood of the accessed historical data point. If the number is less than the minimum number of points, add the accessed historical data point to the cluster set, otherwise add it to the noise point set, to obtain the corresponding cluster set and noise point set.

5. The data anomaly monitoring method according to claim 3, characterized in that: The preprocessing of the historical data includes: Determine whether there is historical abnormal data in the historical data, and if there is historical abnormal data in the historical data, extract the historical abnormal data; According to a preset approximate symbol algorithm, the historical abnormal data is subjected to time series discrete processing and symbolization processing to obtain a symbol sequence corresponding to the historical abnormal data; Calculating the similarity between the symbol sequences according to a preset Euclidean distance algorithm, and then obtaining a data reference curve according to the similarity between the symbol sequences; According to the data reference curve, the time series corresponding to the historical abnormal data is adjusted.

6. The data anomaly monitoring method according to claim 5, characterized in that: The determining whether there is historical abnormal data in the historical data includes: Calculate the median of the historical data; Calculate the lower quartile and the upper quartile of the historical data, and then calculate the interquartile range of the historical data according to the lower quartile and the upper quartile; Calculate the lower outlier threshold and the upper outlier threshold corresponding to the historical data according to the median, the lower quartile, the upper quartile and the interquartile range; The historical data is compared with the lower abnormal value threshold and the upper abnormal value threshold, and the historical data that is less than the lower threshold or greater than the upper threshold is taken as the historical abnormal data.

7. The data anomaly monitoring method according to claim 1, characterized in that: The probability density function value of the data to be monitored is calculated according to the following formula: Among them, n is the number of data features of the data to be monitored; x is the feature vector containing the currently detected power data parameters; μ is the vector of the feature mean; Σ is the covariance matrix; |Σ| is the determinant of the covariance matrix.

8. A data anomaly monitoring device, characterized in that: include: A module for acquiring data to be monitored, a module for calculating probability density function value, a module for identifying abnormal data, and a module for generating abnormal data warning; The module for acquiring data to be monitored is used to acquire the data to be monitored; The probability density function value calculation module is used to input the data to be monitored into a preset multivariate Gaussian distribution model, so that the multivariate Gaussian distribution model calculates the probability density function value of the data to be monitored according to the preset mean and covariance matrix, and outputs the probability density function value of the data to be monitored; wherein the multivariate Gaussian distribution model is obtained by fitting a preset multivariate Gaussian distribution according to historical data; The abnormal data identification module is used to compare the probability density function value with a preset abnormal threshold value, and when the probability density function value is less than the abnormal threshold value, determine that the data to be monitored is abnormal data, otherwise determine that the data to be monitored is normal data; The data anomaly warning generation module is used to generate corresponding data anomaly warning information when it is determined that the data to be monitored is abnormal data.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the data anomaly monitoring method as claimed in any one of claims 1 to 7 when executing the computer program.

10. A storage medium, characterized in that: The storage medium includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the data anomaly monitoring method according to any one of claims 1 to 7.

Images