Data anomaly detection method and device, storage medium and electronic equipment

Abstract

The present application provides a data anomaly detection method and device, a storage medium, and an electronic device. By acquiring a plurality of electricity usage data carrying tagged information and a plurality of electricity usage data not carrying tagged information, a first classifier is trained using the current data in the electricity usage data carrying tagged information, and a second classifier is trained using the power data in the electricity usage data carrying tagged information, thereby using the trained first classifier and the second classifier to perform anomaly detection on the unlabeled electricity usage data, and labeling the electricity usage data not carrying tagged information based on the detection results. It can be seen that the present technical solution only requires manual labeling of part of the electricity usage data, and then training expansion is performed based on the part of the electricity usage data carrying tagged information, automatically and accurately detecting and labeling the remaining unlabeled electricity usage data, thereby reducing labor costs and improving the efficiency and accuracy of data anomaly detection, thereby improving the efficiency and accuracy of labeling.

Description

Technical Field

[0001] The present application relates to the technical field of power data security management, and in particular to a data anomaly detection method and device, a storage medium and an electronic device. Background Art

[0002] Currently, the power system collects and monitors electricity consumption data through sensor equipment, thereby obtaining a large amount of electricity consumption data. However, due to the influence of various factors such as equipment, communication and environment, the collected electricity consumption data may have data quality issues. Therefore, it is necessary to perform anomaly detection on the electricity consumption data and mark whether the electricity consumption data is abnormal based on the anomaly detection results.

[0003] In the existing technology, all electricity consumption data is generally manually judged whether it is abnormal data, and manually marked based on the judgment results. However, due to the large amount of data, a lot of time and manpower costs are consumed, resulting in low anomaly detection efficiency, which in turn leads to low marking efficiency. Summary of the Invention

[0004] The present application provides a data anomaly detection method and device, a storage medium and an electronic device, the purpose of which is to solve the problem that existing solutions require a lot of time and manpower costs, resulting in low detection efficiency and thus low labeling efficiency.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] A data anomaly detection method, comprising:

[0007] Acquire multiple power consumption time series data subsequences; the power consumption time series data subsequences include multiple power consumption data, and the power consumption data is current data or power data;

[0008] For each power consumption time series data subsequence, calculating a correlation coefficient of the power consumption time series data subsequence based on the current data and power data included in the power consumption time series data subsequence;

[0009] Sending each electricity consumption data to the user terminal so that the user can mark part of the electricity consumption data;

[0010] Obtaining a result of feedback from the user terminal; the result includes a plurality of electricity usage data carrying tag information and a plurality of electricity usage data not carrying tag information, wherein the tag information is used to indicate whether the electricity usage data is abnormal data;

[0011] The current data in each power usage data carrying the tag information are grouped into a first set, and the power data in each power usage data carrying the tag information are grouped into a second set;

[0012] The electricity consumption data without tag information are combined into a third set;

[0013] Determining a first classifier and a second classifier from a plurality of preset classifiers;

[0014] Using the first set, training the first classifier;

[0015] Using the second set, training the second classifier;

[0016] Using the trained first classifier to perform anomaly detection on each electricity usage data included in the third set, obtaining a first detection result for each electricity usage data included in the third set; and using the trained second classifier to perform anomaly detection on each electricity usage data included in the third set, obtaining a second detection result for each electricity usage data included in the third set;

[0017] For each electricity usage data included in the third set, calculating a target detection result of the electricity usage data based on the first detection result, the second detection result, and a correlation coefficient of the electricity usage time series data subsequence to which the electricity usage data belongs, and if the target detection result is greater than a confidence threshold, marking the electricity usage data based on the target detection result, adding the marked electricity usage data to the first set and the second set, and deleting the electricity usage sample data from the third set;

[0018] Determine whether the current third set is an empty set. If the current third set is not an empty set, then when the trained second classifier does not meet the stop training condition, based on the current first set and the current second set, return to execute the step of using the first set to train the first classifier until the current third set is an empty set; wherein, the stop training condition is that the correlation coefficients of the power consumption time series data subsequences to which the power consumption data included in the current third set belong are not within a preset range.

[0019] The above method may optionally further include:

[0020] If the trained second classifier meets the training stop condition, the current first classifier is trained using the current first set;

[0021] Using the trained first classifier to perform anomaly detection on each electricity usage data included in the current third set, to obtain a target detection result for each electricity usage data included in the current third set;

[0022] For each electricity usage data included in the third set, if the target detection result is greater than the confidence threshold, marking the electricity usage data based on the target detection result, adding the marked electricity usage data to the first set, and deleting the electricity usage sample data from the third set;

[0023] Determine whether the current third set is an empty set. If the third set is not an empty set, return to the step of training the current first classifier using the current first set based on the current first set until the current third set is an empty set.

[0024] In the above method, optionally, the step of obtaining a subsequence of power consumption time series data includes:

[0025] Acquire a power consumption time series data sequence; the power consumption time series data sequence includes a plurality of power consumption data;

[0026] According to a preset time window, the power consumption time series data sequence is divided into multiple power consumption time series data subsequences.

[0027] The above method may optionally further include, before dividing the power consumption time series data sequence into a plurality of power consumption time series data subsequences according to a preset time window:

[0028] Data preprocessing is performed on each power consumption data included in the power consumption time series data sequence.

[0029] In the above method, optionally, the data preprocessing of each power consumption data included in the power consumption time series data sequence includes:

[0030] Calculating a daily electricity consumption data missing rate based on each electricity consumption data included in the electricity consumption time series data sequence;

[0031] Eliminate each electricity consumption data item whose missing rate is less than a preset missing rate threshold;

[0032] The linear interpolation method is used to complete the electricity consumption data corresponding to the electricity consumption data missing rate that is not less than the preset missing rate threshold.

[0033] Optionally, the method described above includes calculating the target detection result of the electricity usage data based on the first detection result and the second detection result of the electricity usage data and the correlation coefficient of the electricity usage time series data subsequence to which the electricity usage data belongs, including:

[0034] Calculating the product of the second detection result and the correlation coefficient of the power consumption time series data subsequence to which the power consumption data belongs to obtain an initial detection result;

[0035] The first detection result and the initial detection result are summed to obtain a target detection result of the power usage data.

[0036] Optionally, in the above method, determining the first classifier and the second classifier from a plurality of preset classifiers includes:

[0037] Calculate the characteristic statistics of each electricity consumption time series data subsequence based on the electricity consumption data included in each electricity consumption time series data subsequence;

[0038] Based on the respective feature statistics, a first classifier and a second classifier are determined from a plurality of preset classifiers.

[0039] A data anomaly detection device, comprising:

[0040] A first acquiring unit is configured to acquire a plurality of power consumption time series data subsequences, wherein the power consumption time series data subsequences include a plurality of power consumption data, and the power consumption data is current data or power data;

[0041] a calculation unit, configured to calculate, for each power consumption time series data subsequence, a correlation coefficient of the power consumption time series data subsequence based on the current data and power data included in the power consumption time series data subsequence;

[0042] a first marking unit, configured to send each electricity usage data to a user terminal so that the user can mark part of the electricity usage data;

[0043] A second acquiring unit is configured to acquire a result fed back by the user terminal; the result includes a plurality of electricity usage data carrying tag information and a plurality of electricity usage data not carrying tag information, wherein the tag information is used to indicate whether the electricity usage data is abnormal data;

[0044] A first forming unit is configured to form the current data in each power usage data carrying the tag information into a first set, and form the power data in each power usage data carrying the tag information into a second set;

[0045] The second forming unit is configured to form the power consumption data without tag information into a third set;

[0046] a determining unit, configured to determine a first classifier and a second classifier from a plurality of preset classifiers;

[0047] a first training unit, configured to train the first classifier using the first set;

[0048] a second training unit, configured to train the second classifier using the second set;

[0049] a first detection unit, configured to perform anomaly detection on each electricity usage data included in the third set using the trained first classifier to obtain a first detection result for each electricity usage data included in the third set, and to perform anomaly detection on each electricity usage data included in the third set using the trained second classifier to obtain a second detection result for each electricity usage data included in the third set;

[0050] a second marking unit, configured to calculate, for each electricity usage data included in the third set, a target detection result of the electricity usage data based on the first detection result, the second detection result of the electricity usage data, and a correlation coefficient of a subsequence of electricity usage time series data to which the electricity usage data belongs, and, if the target detection result is greater than a confidence threshold, mark the electricity usage data based on the target detection result, add the marked electricity usage data to the first set and the second set, and delete the electricity usage sample data from the third set;

[0051] The first judgment unit is used to determine whether the current third set is an empty set. If the current third set is not an empty set, then when the trained second classifier does not meet the stop training condition, based on the current first set and the current second set, return to execute the step of using the first set to train the first classifier until the current third set is an empty set; wherein, the stop training condition is that the correlation coefficients of the power consumption time series data subsequences to which the power consumption data included in the current third set belong are not within a preset range.

[0052] A storage medium stores an instruction set, wherein the instruction set, when executed by a processor, implements the above-mentioned data anomaly detection method.

[0053] An electronic device, comprising:

[0054] A memory for storing at least one set of instructions;

[0055] The processor is configured to execute the instruction set stored in the memory, and implement the above-mentioned data anomaly detection method by executing the instruction set.

[0056] Compared with the prior art, this application has the following advantages:

[0057] The present application provides a data anomaly detection method and device, a storage medium, and an electronic device. The method includes: obtaining a plurality of electricity usage data carrying tag information and a plurality of electricity usage data not carrying tag information, using the current data in the electricity usage data carrying tag information to train a first classifier, and using the power data in the electricity usage data carrying tag information to train a second classifier, thereby using the trained first classifier and the second classifier to perform anomaly detection on the unlabeled electricity usage data, and marking the electricity usage data not carrying tag information based on the detection results. It can be seen that the present technical solution only requires manual marking of part of the electricity usage data, and then training expansion is performed based on the part of the electricity usage data carrying tags, automatically and accurately realizing anomaly detection and marking of the remaining unlabeled electricity usage data, without the need to manually mark all the electricity usage data, thereby reducing labor costs and improving the efficiency and accuracy of data anomaly detection, thereby improving the efficiency and accuracy of marking. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0059] Figure 1 A flow chart of a data anomaly detection method provided in this application;

[0060] Figure 2 Another method flow chart of a data anomaly detection method provided by this application;

[0061] Figure 3 Another method flow chart of a data anomaly detection method provided by this application;

[0062] Figure 4 Another method flow chart of a data anomaly detection method provided by this application;

[0063] Figure 5 A schematic diagram of the structure of a data anomaly detection device provided in this application;

[0064] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0066] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0067] It should be noted that the concepts of "first" and "second" mentioned in this application are only used to distinguish different systems, modules or units, and are not used to limit the order or interdependence of the functions performed by these systems, modules or units.

[0068] It should be noted that the modifications of "one" and "multiple" mentioned in the disclosure of this application are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0069] The present application can be used in a variety of general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor systems, and distributed computing environments including any of the above systems or devices.

[0070] The present invention provides a method for detecting data anomalies. Figure 1 As shown, specifically including:

[0071] S101: Acquire multiple electricity consumption time series data subsequences.

[0072] In this embodiment, multiple power consumption time series data subsequences are obtained; the power consumption time series data subsequences include multiple power consumption data, and the power consumption data is current data or power data, that is, each power consumption time series data subsequence includes multiple current data and multiple power data.

[0073] For details, see Figure 2 The process of obtaining multiple subsequences of electricity consumption time series data includes the following steps:

[0074] S201. Obtain a time series data sequence of electricity consumption.

[0075] In this embodiment, the power consumption time series data sequence is obtained, specifically, the power consumption data is collected every day according to a preset collection cycle, and the collected power consumption data is formed into a power consumption time series data series. The power consumption data is current data or power data.

[0076] For example, the preset collection period may be to collect data every hour.

[0077] It should be noted that the electricity consumption time series data sequence can be used to obtain the user's electricity consumption pattern by drawing a data curve of the electricity consumption data. For example, the electricity consumption pattern includes the electricity consumption model of going to work in the morning and taking a break in the afternoon, the electricity consumption model of working during the day and taking a break at night, the electricity consumption model of being at home all day, and the electricity consumption model when the house is vacant.

[0078] For example, the data curve from 6:00 to 12:00 is stable, and the data curve from 12:00 to 18:00 is higher than that in the morning time period, which is the electricity consumption pattern of working in the morning and resting in the afternoon.

[0079] S202 : Divide the electricity consumption time series data sequence into multiple electricity consumption time series data subsequences according to a preset time window.

[0080] In this embodiment, the power consumption time series data sequence is divided into multiple power consumption time series data subsequences according to a preset time window, that is, each power consumption data included in the power consumption time series data sequence is divided into multiple power consumption time series data subsequences.

[0081] For example, four hours can be set as a time window, and the 24 hours of a day can be divided into 6:00-12:00 in the morning, 12:00-18:00 in the afternoon, 18:00-24:00 in the first half of the night, and 0:00-6:00 in the second half of the night; twelve hours can also be set as a time window, and the 24 hours of a day can be divided into 6:00-18:00 during the day, 18:00-24:00 in the evening, and 0:00-6:00.

[0082] S102 . For each power consumption time series data subsequence, calculate a correlation coefficient of the power consumption time series data subsequence based on the current data and power data included in the power consumption time series data subsequence.

[0083] In this embodiment, for each power consumption time series data subsequence, based on the current data and power data included in the power consumption time series data subsequence, the correlation coefficient of the current data and the power data is calculated using the correlation coefficient calculation formula, thereby obtaining the correlation coefficient of the power consumption time series data subsequence.

[0084] An exemplary correlation coefficient calculation formula may be a Pearson correlation coefficient calculation formula.

[0085] S103: Send each electricity usage data to the user terminal, so that the user can mark part of the electricity usage data.

[0086] In this embodiment, each electricity usage data included in each electricity usage time series data subsequence is sent to the user terminal, so that the user can mark part of the electricity usage data, that is, mark whether part of the electricity usage data is abnormal.

[0087] It should be noted that users only need to mark a very small amount of electricity consumption data models.

[0088] S104 , obtaining a result fed back by the user terminal; the result includes a plurality of electricity usage data carrying tag information and a plurality of electricity usage data not carrying tag information.

[0089] In this embodiment, after the user completes marking of part of the electricity usage data, the user terminal obtains the result of the loss, which includes multiple electricity usage data with marking information and multiple electricity usage data without marking information. The marking information is used to indicate whether the electricity usage data is abnormal data.

[0090] S105 , grouping the current data in each power usage data carrying tag information into a first set, grouping the power data in each power usage data carrying tag information into a second set, and grouping the power data without tag information into a third set.

[0091] In this embodiment, the current data in each piece of electricity usage data carrying tag information is grouped into a first set, and the power data in each piece of electricity usage data carrying tag information is grouped into a second set.

[0092] In this embodiment, the electricity usage data without tag information are grouped into a third set.

[0093] S106: Determine a first classifier and a second classifier from a plurality of preset classifiers.

[0094] In this embodiment, multiple classifiers are preset, including but not limited to a logistic regression model and a decision tree model.

[0095] In this embodiment, any one classifier is selected from a plurality of preset classifiers as the first classifier, and any one classifier is selected from a plurality of preset classifiers as the second classifier.

[0096] For details, see Figure 3 The process of determining a first classifier and a second classifier from a plurality of preset classifiers includes the following steps:

[0097] S301 : Calculate characteristic statistics of each electricity consumption time series data subsequence based on each electricity consumption data included in each electricity consumption time series data subsequence.

[0098] In this embodiment, based on the individual electricity consumption data included in each electricity consumption time series data subsequence, the characteristic statistical values ​​of each electricity consumption time series data subsequence are calculated. Exemplarily, the characteristic statistical values ​​include but are not limited to the mean, median, standard deviation, maximum value and minimum value.

[0099] S302: Determine a first classifier and a second classifier from a plurality of preset classifiers based on the statistical values ​​of the respective features.

[0100] In this embodiment, based on each feature statistic value, a classifier is selected as the first classifier and a classifier is selected as the second classifier from a plurality of preset classifiers through automatic selection or manual selection.

[0101] Exemplarily, the automatic selection process includes: presetting the correspondence between the numerical interval of the feature statistical value and the classifier, calculating the total feature statistical value based on each feature statistical value, determining the numerical interval to which the total feature statistical value belongs, and then determining the first classifier and the second classifier based on the numerical interval to which it belongs.

[0102] S107: Use the first set to train the first classifier, and use the second set to train the second classifier.

[0103] In this embodiment, the first classifier is trained using the current data included in the first set, or the first classifier is trained using the current data and power data included in the first set.

[0104] If the first classifier is trained for the first time, all data in the first set are current data; if the first classifier is not trained for the first time, the first set includes current data, or the first set includes current data and power data.

[0105] In this embodiment, the second classifier is trained using the power data included in the second set, or the second classifier is trained using the current data and power data included in the second set.

[0106] If the second classifier is trained for the first time, all data in the second set are power data; if the second classifier is not trained for the first time, the second set includes power data, or the first set includes current data and power data.

[0107] S108. Use the trained first classifier to perform anomaly detection on each electricity usage data included in the third set to obtain a first detection result for each electricity usage data included in the third set, and use the trained second classifier to perform anomaly detection on each electricity usage data included in the three sets to obtain a second detection result for each electricity usage data included in the third set.

[0108] In this embodiment, for each electricity consumption data included in the third set, the electricity consumption data is input into the first classifier, and the first classifier performs anomaly detection on the electricity consumption data, that is, detects the probability of whether the electricity consumption data is abnormal data, and obtains the probability of whether the electricity consumption data output by the first classifier is abnormal data, and uses the probability as the first detection result of the electricity consumption data.

[0109] In this embodiment, for each electricity consumption data included in the third set, the electricity consumption data is input into the second classifier, and the second classifier performs anomaly detection on the electricity consumption data, that is, detects the probability of whether the electricity consumption data is abnormal data, and obtains the probability of whether the electricity consumption data output by the second classifier is abnormal data, and uses the probability as the second detection result of the electricity consumption data.

[0110] S109 . For each electricity usage data included in the third set, calculate a target detection result of the electricity usage data based on the first detection result and the second detection result of the electricity usage data and a correlation coefficient of the electricity usage time series data subsequence to which the electricity usage data belongs.

[0111] In this embodiment, for each electricity usage data included in the third set, a target detection result of the electricity usage data is calculated based on the first detection result, the second detection result and the correlation coefficient of the electricity usage time series data subsequence to which the electricity usage data belongs.

[0112] Specifically, the process of calculating the target detection result of the electricity usage data based on the first detection result of the electricity usage data, the second detection result, and the correlation coefficient of the electricity usage time series data subsequence to which the electricity usage data belongs includes the following steps:

[0113] Calculate the product of the second detection result and the correlation coefficient of the power consumption time series data subsequence to which the power consumption data belongs, to obtain an initial detection result;

[0114] The first detection result and the initial detection result are summed to obtain a target detection result of the electricity usage data.

[0115] In this embodiment, the correlation coefficient of the electricity consumption time series data subsequence to which the electricity consumption data belongs is used as the weight of the second classification model, and the product between the second detection result and the correlation coefficient of the electricity consumption time series data subsequence to which the electricity consumption data belongs is calculated to obtain the initial detection result, and the first detection result and the initial detection result are summed to obtain the target detection result of the electricity consumption data.

[0116] The following example illustrates the process of calculating the target detection result of the electricity consumption data based on the first detection result of the electricity consumption data, the second detection result, and the correlation coefficient of the electricity consumption time series data subsequence to which the electricity consumption data belongs:

[0117] The first detection result is a, the second detection result is b, and the correlation coefficient of the electricity consumption time series data subsequence to which the electricity consumption data belongs is k, then the target detection result is a+kb.

[0118] S110. For each electricity usage data included in the third set, when the target detection result is greater than the confidence threshold, mark the electricity usage data based on the target detection result, add the marked electricity usage data to the first set and the second set, and delete the electricity usage sample data from the third set.

[0119] In this embodiment, for each electricity consumption data included in the third and, it is determined whether the target detection result of the electricity consumption data is greater than the confidence threshold. If the target detection result is greater than the confidence threshold, the electricity consumption data is marked based on the target detection result, that is, whether the electricity consumption data is abnormal.

[0120] S111. Determine whether the current third set is an empty set. If not, execute S112. If so, end directly.

[0121] In this embodiment, it is determined whether the current third set is an empty set, that is, it is determined whether all unmarked electricity usage data have been marked.

[0122] S112: Determine whether the trained second classifier meets the training stop condition. If not, return to execute S107 based on the current first set and the current second set. If so, execute S113.

[0123] In this embodiment, if the current third set is not an empty set, it is further determined whether the trained second classifier meets the stop training condition, wherein the stop training condition is that the correlation coefficients of the power consumption time series data subsequences to which the power consumption data included in the current third set belong are not within a preset interval. It can be expressed as η>β1 or η<β2, where η is the correlation coefficient and the preset interval is (β2, β1). That is, if the power data and the current data are completely uncorrelated, then the abnormal pattern of the power data is unrelated to the abnormal pattern of the current data. If the power data and the current data are completely consistent, then the classifier of the power data has little effect on the classifier of the current data.

[0124] In this embodiment, if the trained second classifier does not meet the training stop condition, the process returns to step S108 based on the current first set and the current second set.

[0125] S113: Use the current first set to train the current first classifier.

[0126] In this embodiment, if the trained second classifier meets the training stop condition, the current first set is used to train the current first classifier.

[0127] S114 , using the trained first classifier to perform anomaly detection on each electricity usage data included in the current third set, to obtain a target detection result for each electricity usage data included in the current third set.

[0128] In this embodiment, for each electricity consumption data included in the current third set, the electricity consumption data is input into the first classifier, and the first classifier performs anomaly detection on the electricity consumption data, that is, detects the probability of whether the electricity consumption data is abnormal data, and obtains the probability of whether the electricity consumption data output by the first classifier is abnormal data, and uses the probability as the target detection result of the electricity consumption data.

[0129] S115. For each electricity usage data included in the third set, if the target detection result is greater than the confidence threshold, mark the electricity usage data based on the target detection result, add the marked electricity usage data to the first set, and delete the electricity usage sample data from the third set.

[0130] In this embodiment, for each electricity consumption data included in the third set, it is determined whether the target detection result is greater than the confidence threshold. If the target detection result is greater than the confidence threshold, the electricity consumption data is marked based on the target detection result, and the marked electricity consumption data is added to the first set, and the electricity consumption sample data is deleted from the third set.

[0131] In this embodiment, when the second classifier meets the training stop condition, the label information of the electricity usage data is determined only by the output result of the first classifier.

[0132] S116. Determine whether the current third set is an empty set. If not, return to execute S113 based on the current first set. If so, terminate directly.

[0133] In this embodiment, it is determined whether the current third set is an empty set. If the current third set is an empty set, it means that all electricity usage data have been marked. If the current third set is not an empty set, it means that there is still unmarked electricity usage data, and based on the current first set, return to execute step S115 until the current third set is an empty set.

[0134] It should be noted that, in this embodiment, the idea of ​​the acquisition collaborative training method is combined with the self-training method to improve the accuracy of the marking of electricity consumption data.

[0135] The data anomaly detection method provided in the embodiment of the present application obtains a plurality of electricity usage data carrying tagged information and a plurality of electricity usage data not carrying tagged information, uses the current data in the electricity usage data carrying tagged information to train a first classifier, and uses the power data in the electricity usage data carrying tagged information to train a second classifier, thereby using the trained first classifier and the second classifier to perform anomaly detection on the unlabeled electricity usage data, and marking the electricity usage data not carrying tagged information based on the detection results. It can be seen that the present technical solution only requires manual marking of part of the electricity usage data, and then performs training expansion based on the part of the electricity usage data carrying the tagged information, automatically and accurately realizing anomaly detection and marking of the remaining unlabeled electricity usage data, without the need to manually mark all the electricity usage data, thereby reducing labor costs and improving the efficiency and accuracy of data anomaly detection, thereby improving the efficiency and accuracy of marking.

[0136] In this embodiment, the data anomaly detection method provided in the embodiment of the present application may further include the following steps before step S202:

[0137] Data preprocessing is performed on each power consumption data included in the power consumption time series data sequence.

[0138] In this embodiment, before dividing the power consumption time series data sequence into multiple power consumption time series data subsequences according to the preset time window, data preprocessing can also be performed on each power consumption data included in the power consumption time series data sequence. Figure 4 , specifically including the following steps:

[0139] S401. Calculate the daily electricity consumption data missing rate based on each electricity consumption data included in the electricity consumption time series data sequence.

[0140] In this embodiment, the daily electricity usage data missing rate is calculated based on the electricity usage data included in the electricity usage time series data sequence. Specifically, the electricity usage data included in the electricity usage time series data sequence is divided into multiple sets according to the date corresponding to the electricity usage data. Each set includes electricity usage data for one day. For each set, the data missing rate of the set is calculated to obtain the daily electricity usage data missing rate.

[0141] S402 : For the daily electricity consumption data missing rate, determine whether the electricity consumption data missing rate is less than a preset missing rate threshold. If so, execute S403 ; otherwise, execute S404 .

[0142] For the daily electricity consumption data missing rate, determine whether the electricity consumption data missing rate is less than a preset missing rate threshold. Preferably, the preset missing rate threshold is 20%.

[0143] S403: Use a linear interpolation method to complete the electricity consumption data corresponding to the missing rate of electricity consumption data.

[0144] In this embodiment, for each day's missing electricity data rate, if the missing electricity data rate is less than a preset missing data rate threshold, linear interpolation is used to complete the electricity data corresponding to the missing electricity data rate. Specifically, for each missing electricity data point, the average of the previous and next electricity data points is calculated, and this average is used as the missing electricity data point.

[0145] S404: Eliminate each electricity consumption data corresponding to the electricity consumption data missing rate.

[0146] In this embodiment, for the daily electricity consumption data missing rate, if the electricity consumption data missing rate is not less than a preset missing rate threshold, each electricity consumption data corresponding to the electricity consumption data missing rate is eliminated, that is, all electricity consumption data for that day is eliminated.

[0147] In the data anomaly detection method provided in the embodiment of the present application, the data quality of the original electricity usage data is improved by performing data preprocessing on each electricity usage data included in the electricity usage time series data sequence.

[0148] The specific implementation process of the data anomaly detection method provided in the embodiment of the present application is illustrated as follows:

[0149] (1) Perform multi-dimensional vector data preprocessing based on time window for electricity consumption collection time series data.

[0150] The multidimensional vector data preprocessing based on the time window includes:

[0151] Set the time window Δt. Optionally, the time window can be set to 4 according to the daily electricity consumption periodicity, that is, the 24 hours a day is divided into 6:00-12:00 in the morning, 12:00-18:00 in the afternoon, 18:00-24:00 in the first half of the night, and 0:00-6:00 in the second half of the night; it can also be set to 2, that is, the 24 hours a day is divided into 6:00-18:00 during the day, 18:00-24:00 in the evening, and 0:00-6:00.

[0152] Severely missing data with a missing percentage of 20% or more were eliminated, while data with a missing percentage less than 20% were supplemented with the help of linear interpolation.

[0153] Convert the user current and power acquisition data into multi-dimensional vector data XΔt=(x t1 ,x t2 ,……,x tk ) and X'Δt=(x t1 ,x t2 ,……,x tk), k is the collected data in the time period, and each XΔt, X'Δt implies the electricity consumption pattern of the user during this period.

[0154] Calculate the time series feature statistics of each user's multidimensional vector data within the sliding window, optionally including the mean, median, standard deviation, maximum, minimum, etc. of the time period data.

[0155] (2) Select one or more models through the model selector, use the labeled samples to train the classifier, assign different weights through one or more classifiers, select unlabeled samples with high confidence, and add them to the next round of training.

[0156] The model selector includes a series of basic algorithm models, such as logistic regression model (LR) and decision tree model (LightGBM), etc., and selects a suitable basic algorithm model through feature statistics in preprocessing.

[0157] Initialize the labeled current data and power data, select models for training respectively, and generate model training classifiers respectively.

[0158] Calculate the correlation coefficient between current data and power data within the sliding window.

[0159] Classifiers are trained based on the two models respectively, and the labeling probability of unlabeled data is calculated.

[0160] The weight of the labeling of the unlabeled data by the model classifier based on power data is the correlation coefficient between the current data and the power data.

[0161] The labeling probability and threshold of the two model classifiers jointly determine the labeling results of the current data.

[0162] (3) Determine whether the classifier needs to be retrained through the training judge.

[0163] Optionally, if the correlation coefficient between the current data and the power data in each sliding window is outside a threshold, the model classifier based on the power data is not retrained, and the labeling result is determined only by the classifier of the current data.

[0164] (4) The termination judge determines whether the stopping condition is met and outputs the final result.

[0165] The termination condition is that the data sample set whose label probability output by the model classifier based on the current data is greater than the threshold is empty.

[0166] It should be noted that although the operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order. In certain circumstances, multitasking and parallel processing may be advantageous.

[0167] It should be understood that the various steps described in the method embodiments disclosed herein may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0168] and Figure 1 Corresponding to the method described above, the embodiment of the present application further provides a data anomaly detection device for detecting Figure 1 The specific implementation of the method is shown in the following diagram: Figure 5 As shown, specifically including:

[0169] A first acquiring unit 501 is configured to acquire a plurality of power consumption time series data subsequences, wherein the power consumption time series data subsequences include a plurality of power consumption data, and the power consumption data is current data or power data;

[0170] a calculation unit 502 configured to calculate, for each power consumption time series data subsequence, a correlation coefficient of the power consumption time series data subsequence based on the current data and power data included in the power consumption time series data subsequence;

[0171] The first marking unit 503 is used to send each power consumption data to the user terminal so that the user can mark part of the power consumption data;

[0172] The second acquisition unit 504 is configured to acquire a result fed back by the user terminal; the result includes a plurality of electricity usage data carrying tag information and a plurality of electricity usage data not carrying tag information, wherein the tag information is used to indicate whether the electricity usage data is abnormal data;

[0173] A first grouping unit 505 is configured to group the current data in each power usage data carrying the tag information into a first set, and group the power data in each power usage data carrying the tag information into a second set;

[0174] The second forming unit 506 is configured to form the power usage data without tag information into a third set;

[0175] a determining unit 507, configured to determine a first classifier and a second classifier from a plurality of preset classifiers;

[0176] A first training unit 508 is configured to train the first classifier using the first set;

[0177] A second training unit 509 is configured to train the second classifier using the second set;

[0178] a first detection unit 510 configured to perform anomaly detection on each electricity usage data included in the third set using the trained first classifier to obtain a first detection result for each electricity usage data included in the third set, and to perform anomaly detection on each electricity usage data included in the third set using the trained second classifier to obtain a second detection result for each electricity usage data included in the third set;

[0179] a second marking unit 511 configured to calculate, for each electricity usage data included in the third set, a target detection result of the electricity usage data based on the first detection result, the second detection result, and a correlation coefficient of the electricity usage time series data subsequence to which the electricity usage data belongs, and, if the target detection result is greater than a confidence threshold, mark the electricity usage data based on the target detection result, add the marked electricity usage data to the first set and the second set, and delete the electricity usage sample data from the third set;

[0180] The first judgment unit 512 is used to determine whether the current third set is an empty set. If the current third set is not an empty set, then when the trained second classifier does not meet the stop training condition, based on the current first set and the current second set, it returns to execute the step of using the first set to train the first classifier until the current third set is an empty set; wherein, the stop training condition is that the correlation coefficients of the power consumption time series data subsequences to which the power consumption data included in the current third set belong are not within the preset range.

[0181] In the data anomaly detection device provided in the embodiment of the present application, it is only necessary to manually mark part of the electricity usage data, so as to perform training expansion based on the marked electricity usage data, and automatically and accurately realize anomaly detection and marking of the remaining unlabeled electricity usage data, without the need to manually mark all the electricity usage data, thereby reducing labor costs and improving the efficiency and accuracy of data anomaly detection, thereby improving the efficiency and accuracy of marking.

[0182] In one embodiment of the present application, based on the above solution, it can also be configured as follows:

[0183] a third training unit, configured to train the current first classifier using the current first set if the trained second classifier meets a training stop condition;

[0184] a second detection unit, configured to perform anomaly detection on each electricity usage data included in the current third set using the trained first classifier, and obtain a target detection result for each electricity usage data included in the current third set;

[0185] a third marking unit, configured to, for each electricity usage data included in the third set, mark the electricity usage data based on the target detection result if the target detection result is greater than the confidence threshold, add the marked electricity usage data to the first set, and delete the electricity usage sample data from the third set;

[0186] The second judgment unit is used to judge whether the current third set is an empty set. If the third set is not an empty set, based on the current first set, it returns to execute the step of using the current first set to train the current first classifier until the current third set is an empty set.

[0187] In one embodiment of the present application, based on the above solution, the first acquiring unit 501 is specifically configured to:

[0188] Acquire a power consumption time series data sequence; the power consumption time series data sequence includes a plurality of power consumption data;

[0189] According to a preset time window, the power consumption time series data sequence is divided into multiple power consumption time series data subsequences.

[0190] In one embodiment of the present application, based on the above solution, it can also be configured as follows:

[0191] The preprocessing unit is used to perform data preprocessing on each power consumption data included in the power consumption time series data sequence.

[0192] In one embodiment of the present application, based on the above solution, the pre-processing unit is specifically configured to:

[0193] Calculating a daily electricity consumption data missing rate based on each electricity consumption data included in the electricity consumption time series data sequence;

[0194] Eliminate each electricity consumption data item whose missing rate is less than a preset missing rate threshold;

[0195] The linear interpolation method is used to complete the electricity consumption data corresponding to the electricity consumption data missing rate that is not less than the preset missing rate threshold.

[0196] In one embodiment of the present application, based on the aforementioned solution, the second marking unit 511, when calculating the target detection result of the power consumption data based on the first detection result of the power consumption data, the second detection result, and the correlation coefficient of the power consumption time series data subsequence to which the power consumption data belongs, is specifically configured to:

[0197] Calculating the product of the second detection result and the correlation coefficient of the power consumption time series data subsequence to which the power consumption data belongs to obtain an initial detection result;

[0198] The first detection result and the initial detection result are summed to obtain a target detection result of the power usage data.

[0199] In one embodiment of the present application, based on the above solution, the determining unit 507 is specifically configured to:

[0200] Calculate the characteristic statistics of each electricity consumption time series data subsequence based on the electricity consumption data included in each electricity consumption time series data subsequence;

[0201] Based on the respective feature statistics, a first classifier and a second classifier are determined from a plurality of preset classifiers.

[0202] An embodiment of the present application further provides a storage medium storing an instruction set, wherein when the instruction set is executed, the data anomaly detection method disclosed in any of the above embodiments is executed.

[0203] The present application also provides an electronic device, the structure of which is shown in FIG. Figure 6 As shown, it specifically includes a memory 601 for storing at least one set of instruction sets; a processor 602 for executing the instruction set stored in the memory, and implementing the data anomaly detection method disclosed in any of the above embodiments by executing the instruction set.

[0204] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

[0205] Although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features described in the context of a separate embodiment may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0206] The above description is merely an illustration of the preferred embodiments disclosed in this application and the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A data anomaly detection method, characterized in that: include: Obtaining multiple electricity consumption time series data subsequences; The power consumption time series data subsequence includes a plurality of power consumption data, and the power consumption data is current data or power data; For each power consumption time series data subsequence, calculating a correlation coefficient of the power consumption time series data subsequence based on the current data and power data included in the power consumption time series data subsequence; Sending each electricity consumption data to the user terminal so that the user can mark part of the electricity consumption data; Obtaining a result of feedback from the user terminal; the result includes a plurality of electricity usage data carrying tag information and a plurality of electricity usage data not carrying tag information, wherein the tag information is used to indicate whether the electricity usage data is abnormal data; The current data in each power usage data carrying the tag information are grouped into a first set, and the power data in each power usage data carrying the tag information are grouped into a second set; The electricity consumption data without tag information are combined into a third set; Determining a first classifier and a second classifier from a plurality of preset classifiers; Using the first set, training the first classifier; Using the second set, training the second classifier; Using the trained first classifier to perform anomaly detection on each electricity usage data included in the third set, obtaining a first detection result for each electricity usage data included in the third set; and using the trained second classifier to perform anomaly detection on each electricity usage data included in the third set, obtaining a second detection result for each electricity usage data included in the third set; For each electricity usage data included in the third set, calculating a target detection result of the electricity usage data based on the first detection result, the second detection result, and a correlation coefficient of the electricity usage time series data subsequence to which the electricity usage data belongs, and if the target detection result is greater than a confidence threshold, marking the electricity usage data based on the target detection result, adding the marked electricity usage data to the first set and the second set, and deleting the electricity usage sample data from the third set; Determining whether the current third set is an empty set; if the current third set is not an empty set, then, when the trained second classifier does not meet the stop training condition, returning to the steps of training the first classifier using the first set and training the second classifier using the second set based on the current first set and the current second set, until the current third set is an empty set; wherein the stop training condition is that the correlation coefficients of the power consumption time series data subsequences to which the power consumption data included in the current third set belong are all outside a preset range; If the current third set is not an empty set and the trained second classifier meets the training stop condition, the current first set is used to train the current first classifier; Using the trained first classifier to perform anomaly detection on each electricity usage data included in the current third set, to obtain a target detection result for each electricity usage data included in the current third set; For each electricity usage data included in the third set, if the target detection result is greater than the confidence threshold, marking the electricity usage data based on the target detection result, adding the marked electricity usage data to the first set, and deleting the electricity usage sample data from the third set; Determine whether the current third set is an empty set. If the third set is not an empty set, return to the step of training the current first classifier using the current first set based on the current first set until the current third set is an empty set.

2. The method according to claim 1, characterized in that The obtaining of a subsequence of power consumption time series data includes: Acquire a power consumption time series data sequence; the power consumption time series data sequence includes a plurality of power consumption data; According to a preset time window, the power consumption time series data sequence is divided into multiple power consumption time series data subsequences.

3. The method according to claim 2, characterized in that Before dividing the power consumption time series data sequence into a plurality of power consumption time series data subsequences according to a preset time window, the method further includes: Data preprocessing is performed on each power consumption data included in the power consumption time series data sequence.

4. The method according to claim 3, characterized in that The performing data preprocessing on each power consumption data included in the power consumption time series data sequence includes: Calculating a daily electricity consumption data missing rate based on each electricity consumption data included in the electricity consumption time series data sequence; Eliminate each electricity consumption data item whose missing rate is less than a preset missing rate threshold; The linear interpolation method is used to complete the electricity consumption data corresponding to the electricity consumption data missing rate that is not less than the preset missing rate threshold.

5. The method according to claim 1, wherein The calculating the target detection result of the electricity usage data based on the first detection result and the second detection result of the electricity usage data and the correlation coefficient of the electricity usage time series data subsequence to which the electricity usage data belongs includes: Calculating the product of the second detection result and the correlation coefficient of the power consumption time series data subsequence to which the power consumption data belongs to obtain an initial detection result; The first detection result and the initial detection result are summed to obtain a target detection result of the power usage data.

6. The method according to claim 1, characterized in that Determining the first classifier and the second classifier from a plurality of preset classifiers includes: Calculate the characteristic statistics of each electricity consumption time series data subsequence based on the electricity consumption data included in each electricity consumption time series data subsequence; Based on the respective feature statistics, a first classifier and a second classifier are determined from a plurality of preset classifiers.

7. A data anomaly detection device, characterized in that: include: A first acquiring unit is configured to acquire a plurality of power consumption time series data subsequences; The power consumption time series data subsequence includes a plurality of power consumption data, and the power consumption data is current data or power data; a calculation unit, configured to calculate, for each power consumption time series data subsequence, a correlation coefficient of the power consumption time series data subsequence based on the current data and power data included in the power consumption time series data subsequence; a first marking unit, configured to send each electricity usage data to a user terminal so that the user can mark part of the electricity usage data; A second acquiring unit is configured to acquire a result fed back by the user terminal; the result includes a plurality of electricity usage data carrying tag information and a plurality of electricity usage data not carrying tag information, wherein the tag information is used to indicate whether the electricity usage data is abnormal data; A first forming unit is configured to form the current data in each power usage data carrying the tag information into a first set, and form the power data in each power usage data carrying the tag information into a second set; The second forming unit is configured to form the power consumption data without tag information into a third set; a determining unit, configured to determine a first classifier and a second classifier from a plurality of preset classifiers; a first training unit, configured to train the first classifier using the first set; a second training unit, configured to train the second classifier using the second set; a first detection unit, configured to perform anomaly detection on each electricity usage data included in the third set using the trained first classifier to obtain a first detection result for each electricity usage data included in the third set, and to perform anomaly detection on each electricity usage data included in the third set using the trained second classifier to obtain a second detection result for each electricity usage data included in the third set; a second marking unit, configured to calculate, for each electricity usage data included in the third set, a target detection result of the electricity usage data based on the first detection result, the second detection result of the electricity usage data, and a correlation coefficient of a subsequence of electricity usage time series data to which the electricity usage data belongs, and, if the target detection result is greater than a confidence threshold, mark the electricity usage data based on the target detection result, add the marked electricity usage data to the first set and the second set, and delete the electricity usage sample data from the third set; a first judging unit, configured to judge whether the current third set is an empty set; if the current third set is not an empty set, then when the trained second classifier does not meet a stop training condition, returning to execute the first training unit and the second training unit based on the current first set and the current second set until the current third set is an empty set; wherein the stop training condition is that the correlation coefficients of the power consumption time series data subsequences to which the power consumption data included in the current third set belong are all outside a preset range; a third training unit, configured to train the current first classifier using the current first set if the current third set is not an empty set and the trained second classifier meets a training stop condition; a second detection unit, configured to perform anomaly detection on each electricity usage data included in the current third set using the trained first classifier, and obtain a target detection result for each electricity usage data included in the current third set; a third marking unit, configured to, for each electricity usage data included in the third set, mark the electricity usage data based on the target detection result if the target detection result is greater than the confidence threshold, add the marked electricity usage data to the first set, and delete the electricity usage sample data from the third set; The second judgment unit is used to determine whether the current third set is an empty set. If the third set is not an empty set, based on the current first set, it returns to execute the third training unit and uses the current first set to train the current first classifier until the current third set is an empty set.

8. A storage medium, characterized in that: The storage medium stores an instruction set, wherein the instruction set, when executed by a processor, implements the data anomaly detection method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: include: A memory for storing at least one set of instructions; A processor is configured to execute an instruction set stored in the memory, and implement the data anomaly detection method according to any one of claims 1 to 6 by executing the instruction set.

Images