Electric power and electric quantity anomaly detection method based on big data

Abstract

The invention discloses an electric power and electric quantity anomaly detection method based on big data, relates to the technical field of electric power and electric quantity anomaly detection, and provides online concept drift detection, model adaptive updating, group anomaly judgment and multi-scale fusion recognition for the problems of high false alarm and missing alarm caused by dynamic change of power consumption behaviors of power consumers. Potential mode changes are found through a sliding window and a threshold value, and a suspected exception list is generated; correcting the short-term model by using incremental training and periodically updating the long-term model; then, multi-scale error measurement is fused into a multi-scale anomaly score, and key anomalies and conventional anomalies are judged in combination with a group deviation rate; and finally, self-adaptive adjustment is carried out through a threshold correction index and the like, and a manual annotation result is returned to the model library, so that accurate distinguishing and continuous learning of normal behavior transformation and real abnormity are realized. According to the scheme, the false alarm rate and the missing report rate can be remarkably reduced, and the reliability and the refinement degree of power dispatching and monitoring are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power and electricity anomaly detection, and specifically to a method for power and electricity anomaly detection based on big data. Background Art

[0002] In the big data environment of electricity consumption in the contemporary power industry, the load behaviors of various users show a high degree of dynamics and diversity. Taking residential users as an example, due to factors such as changes in family structure, new electrical appliances (such as electric vehicles, smart air-conditioning equipment, etc.), and festival and season transitions, the load curve will fluctuate significantly within a short period, and the electricity consumption during the night and peak hours may suddenly increase or decrease at any time. Industrial and commercial users also face dynamic electricity demands. When the production line expands or decreases, the overall energy consumption level may also experience a sharp jump in the short term. In addition, in local areas or special periods (such as during high temperature, severe cold, or public health events), the group electricity consumption behavior will show an overall increase or decrease, forming a load pattern deviation with a wide range and multiple time scales. With the continuous upgrade of the power metering and dispatching network, these massive electricity consumption data are being collected in real time and applied to operation monitoring, energy efficiency management, and intelligent dispatching. However, since the user load behavior is not constant, if the normal behavior drift and real abnormal events cannot be distinguished in time, the traditional anomaly detection methods based on fixed thresholds or historical patterns are prone to a large number of misjudgments in actual operation and maintenance, resulting in management problems such as waste of dispatching resources and delay in troubleshooting.

[0003] In response to the above load dynamic characteristics, existing power anomaly detection systems often use the fixed model parameters or empirical thresholds obtained from early training, lacking the ability of adaptive learning for concept drift, resulting in a large number of normal changes in real scenarios being misreported as anomalies by the system. At the same time, there may be missed reports for progressive electricity theft behaviors or equipment failures. Traditional methods mainly rely on initial historical data to establish a "normal" electricity consumption pattern and default that the future distribution is relatively stable, but ignore the distribution mutations caused by users' new high-power equipment, season transitions, etc.; even if some systems adopt regular manual updates or retrain the model during the annual inspection period, they often cannot respond in time to the frequent adjustments of user behavior due to lag. When the load curve rises or falls significantly within a short period, the static threshold cannot accurately capture the difference between "caused by normal changes" and "real abnormal events", resulting in an uncontrollable soaring false alarm rate. At the same time, abnormal samples are usually scarce and diverse, and supervised learning faces the challenge of unbalanced distribution, resulting in insensitivity to newly emerging abnormal types.

[0004] In summary, this technical deficiency of lacking adaptive response to concept drift and being difficult to distinguish real anomalies from normal mode gradual changes has become the main bottleneck for grid intelligent monitoring and efficient operation and maintenance. Summary of the Invention

[0005] (1) Technical problems to be solved In view of the deficiencies of the prior art, the present invention provides a method for detecting abnormal electric power and electricity based on big data. By using a sliding window and a threshold, potential pattern changes are discovered and a list of suspected anomalies is generated; incremental training is used to correct the short-term model and the long-term model is updated periodically; subsequently, multi-scale error metrics are fused into multi-scale anomaly scores, and key anomalies and normal anomalies are determined by combining the population deviation rate; finally, adaptive adjustment is performed through threshold correction indicators, etc., and the manually labeled results are transmitted back to the model library to achieve accurate discrimination and continuous learning of normal behavior changes and real anomalies, significantly reducing the false alarm and missed alarm rates, and improving the reliability and refinement of power dispatching and monitoring, thereby solving the technical problems described in the background art.

[0006] (2) Technical solutions To achieve the above objectives, the present invention is realized through the following technical solutions: A method for detecting abnormal electric power and electricity based on big data, including When there is a significant deviation between the load distribution and the historical prediction results, online concept drift detection is triggered. A sliding window and a threshold are used to monitor and calculate the logarithmic hyperbolic error collaboratively to identify sudden distribution changes, mark potential pattern changes, and generate a list of suspected anomalies; After receiving the potential pattern change mark and the list of suspected anomalies, model adaptive update is triggered. Incremental training is used to correct the short-term model and the long-term model is trained periodically to generate a new prediction baseline and incorporate the latest normal load pattern; When a new load record is obtained, group anomaly determination and multi-scale fusion recognition are performed. The long-term model and the short-term model are used to calculate the error metrics respectively and aggregated to generate multi-scale anomaly scores, and then a list of key anomalies and normal anomalies is generated by comparing with the multi-scale anomaly threshold; After receiving the list of key anomalies and normal anomalies, human-machine collaborative feedback and continuous optimization iteration are triggered. The manually labeled results are used to correct the drift determination threshold and the multi-scale anomaly threshold, and the labeled samples are injected into the incremental training pipeline to complete closed-loop self-learning.

[0007] Preferably, the load distribution and reference data within the detection window are collected, and the logarithmic stretching deviation at each moment is analyzed; A weighted accumulation strategy is introduced to obtain the comprehensive deviation value at the current moment. When and only when the comprehensive deviation value is greater than the drift determination threshold, the data corresponding to the period is marked as a potential pattern change. Otherwise, only the normal anomaly records are saved in the normal anomaly list.

[0008] Preferably, for each load record in the load record set, a preliminary anomaly determination is formed by comparing the difference with the existing short-term or long-term prediction values according to the preset business rules; Mark the records with significant differences with suspected anomaly tags, pack all suspected anomaly records into a suspected anomaly list, and output the possible drift period markers.

[0009] Preferably, collect the data records that intersect with the pattern change markers and the suspected anomaly list in multiple recent windows to form a short-term training set. After using the logarithmic hyperbolic cost as the objective function for incremental learning, iterate and update the parameter vector to be learned according to gradient descent or its variants. After the iteration converges, obtain the updated parameters of the short-term model to form a new short-term prediction or discrimination baseline.

[0010] Preferably, set a training set that only retains normal and confirmed transformative load data, define the periodic smoothed logarithmic hyperbolic cost as the training objective of the long-term model, and perform batch or mini-batch optimization on the periodic smoothed logarithmic hyperbolic cost.

[0011] Preferably, for the power load data from the current period, apply the short-term model and the long-term model respectively to calculate the multi-scale anomaly scores for each record; For each record, if the multi-scale anomaly score is higher than the anomaly threshold, mark it as a potential anomaly record; if the multi-scale anomaly score is within the preset gray area range, it can be rechecked later.

[0012] Preferably, for the records of each user, if the multi-scale anomaly score is higher than the anomaly threshold and the long-term and short-term errors are both high, pre-include it in the potential individual anomaly list, otherwise regard it as a record at the normal level or in the gray area; Construct a population deviation rate to identify whether a large number of users show trends of increasing or decreasing load at the same time, so as to distinguish between overall pattern transfer and individual anomalies, and generate lists of key anomalies and regular anomalies.

[0013] Preferably, conduct a review on the generated lists of key anomalies and regular anomalies, output the annotation information and store it in the annotation library; for the records with minor errors or obvious group characteristics in the regular anomaly list, process them in batches and mark them as false alarms or real anomalies.

[0014] Preferably, if the short-term model and the long-term model find newly emerging real anomalies or non-anomaly samples, perform momentum or incremental updates on the drift determination threshold and the multi-scale anomaly threshold according to the new annotation information, and send the confirmed normal or anomaly samples back to the model training pipeline in the second step.

[0015] Preferably, when the threshold correction value constructed through integration and exponential amplification operations continues to be greater than the expectation for a period of time, increase or decrease the values of the drift determination threshold and the multi-scale anomaly threshold. If the threshold correction value remains stable or lower than the expectation, maintain the current settings; Samples that have been marked as false positives but were previously classified as anomalies by short-term or long-term models are re-included in the normal training set. True anomaly samples are included in the anomaly training set, and uncertain samples are temporarily stored in the candidate set.

[0016] (III) Beneficial Effects The present invention provides a method for detecting power quantity anomalies based on big data, which has the following beneficial effects: Using the drift determination threshold Combined with a sliding window and real-time monitoring, when significant deviations occur, relevant data are marked as potential pattern changes and a list of suspected anomalies is output, ensuring timely capture of sudden and trend changes.

[0017] For potential drift data, the short-term model and the long-term model are adaptively updated by means of incremental learning or sliding window training. New load patterns confirmed to be normal are incorporated into the normal baseline, greatly enhancing the robustness in dealing with situations such as seasonal changes and newly added high-power equipment.

[0018] Using the short-term error and the long-term error to construct a multi-scale anomaly score , and under the analysis combined with the group deviation rate, the anomaly level is subdivided according to the anomaly threshold : If most users show a simultaneous upward load trend in the same direction, it is determined as a regular anomaly or a group pattern change; otherwise, individual deviations are included in the key anomaly list, significantly reducing concentrated false positives.

[0019] After obtaining the key anomalies and regular anomalies, through review and interactive annotation, the label results are output, and are fed back to the drift determination threshold and the multi-scale anomaly threshold for dynamic tuning, and new positive and negative samples are provided for the short-term and long-term models. The formed closed-loop self-learning mechanism can continuously accumulate anomaly patterns in real scenarios, inhibit the cumulative effect of false positives and missed reports, and also maintain sufficient sensitivity to a small number of high-risk anomalies. Description of the Drawings

[0020] Figure 1 It is a schematic flow diagram of the method for detecting power quantity anomalies based on big data of the present invention. Detailed Embodiments

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0022] Please refer to Figure 1 , the present invention provides a method for detecting abnormal power consumption based on big data, including, Step 1: When a significant difference is detected between the current period load distribution and the historical model prediction result, perform online concept drift detection and preliminary screening for real-time or quasi-real-time data Adopt a sliding window and threshold monitoring, and judge the statistical deviation. Then, implement segmented alarm screening according to the degree of threshold overrun, output potential pattern changes and mark the suspected abnormal list, and store it in the monitoring database; The content of Step 1 is as follows: Step 101: Weighted distribution deviation measure Introduce a custom weighted distribution deviation measure function, and determine whether concept drift occurs by monitoring the load distribution within the real-time / quasi-real-time monitoring window. The specific technical logic is as follows: Let represent the actual load distribution vector at time (or discrete batch ), which contains multiple components representing electricity consumption characteristics; let represent the reference distribution vector corresponding to time , which can be given by a pre-trained historical model or prediction module; Set the sliding window length to , used to collect distribution deviation information for the last time periods; select an attenuation factor , used to give a decreasing weight to the deviation contribution of earlier time periods during the accumulation process; at time, for the data within the window , calculate the logarithmically stretched deviation shown in the following formula for each time:

[0023] where: is the dimension number (i.e., the number of components) of the load distribution vector ; is the deviation stretching coefficient, with a value greater than 0, used to amplify or suppress the sensitivity of the distribution difference; is a small constant to prevent the denominator from being zero; and are the th load characteristics at time The actual value and the reference value; To make the detection more sensitive to recent deviations, a weighted accumulation strategy is introduced to obtain the comprehensive deviation value at the current moment :

[0024] Set a drift determination threshold , and when and only when the comprehensive deviation value is greater than the drift determination threshold , it is determined that significant concept drift may have occurred within the window segment corresponding to the moment , triggering subsequent preliminary screening; When in use, it can timely detect obvious shifts in the user load distribution over time, avoid misjudgment in subsequent processes caused by sudden fluctuations in a single scenario (such as the turning on of high-power equipment of a certain user), and provide highly targeted information on possible drift time periods, reducing the repeated calculation of the undifferentiated interval and improving the overall detection efficiency.

[0025] Step 102, Preliminary anomaly screening and data marking According to the obtained determination results of possible drift time periods, conduct refined preliminary anomaly screening on the load data of the corresponding time periods and output a list of suspected anomalies: If the drift threshold is not reached, only retain the regular anomaly records, as follows: Obtain the possible drift time period mark: that is, when , for the corresponding moment and the section within its sliding window, automatically mark the data of this time period as a potential pattern change mark ; Let represent the set of all load records within this section; If , it is considered that no obvious concept drift is detected, and only the regular anomaly records are saved to the regular anomaly list; for each load record in the load record set , conduct a difference comparison with the existing short-term or long-term prediction values, and form a preliminary anomaly determination according to the preset business rules (such as the upper and lower limits of basic power, rationality of electricity consumption time periods, etc.); mark the records with significant differences (exceeding the set business rule threshold) with suspected anomaly labels, and store them in association with the corresponding possible drift time periods, and package all suspected anomaly records into a list of suspected anomalies , and output the possible drift time period mark; When in use, utilize the load record set to achieve targeted screening, avoid comparing each piece of massive data one by one, improve the processing performance, and output the list of suspected anomalies Closely associated with the possible drift period, it provides high-value clues for the subsequent model update stage, enabling the new and old baselines to be treated differently. It can significantly reduce the ineffective operations in the non-differentiated interval in the subsequent steps. By defining the possible drift period, it realizes the centralized utilization of resources and outputs the results of the preliminary screening in a structured manner, ensuring that in the next step, more in-depth model adaptation and feature correction can be carried out on the suspected abnormal data, greatly improving the accuracy and efficiency of the subsequent processing.

[0026] Step 2: After receiving the potential pattern change markers and the list of suspected anomalies, perform model adaptive update and multi-time scale maintenance for the short-term model parameters Use incremental training to handle recent sudden changes, and for the long-term model parameters Adopt periodic training to learn seasonal trends, respectively incorporate the confirmed normal behaviors or retain the abnormal feature labels, and output new prediction baselines through the updated short-term and long-term models; The above Step 2 includes the following contents: Step 201: Incremental update of the short-term model For the potential pattern change markers and the list of suspected anomalies Perform incremental update of the short-term model. The core is to quickly correct the model parameters by means of the load data in the recent period, so that the model can adapt to the possible sudden changes in user electricity consumption in a timely manner; the pattern change markers represent the concept drift periods detected in Step 1. The load data in these periods has a high probability of representing new behavior patterns; are the suspected abnormal data in the same period or adjacent periods. If some of them are confirmed to be normal in the subsequent process, they can be regarded as samples of short-term pattern changes; Let represent the load feature vector (including multi-dimensional features such as the power of different electrical appliances and time periods) at time (or discrete batch ); Let represent the actual load amount (or corresponding monitoring value) at time , which is used as the learning target of the short-term model; Let represent the output function of the short-term model, be the parameter vector to be learned; Select the window size , collect the data records within the most recent moments that intersect with the pattern change markers and the list of suspected anomalies to form the short-term training set , and define the following logarithmic hyperbolic cost as the objective function of incremental learning:

[0027] Where: is the time decay factor, so that the data closer to the current one has a higher weight in incremental optimization; It is a hyperbolic cosine function, which can maintain smoothness when the error is small and provide stronger penalties when the error is large, thus taking into account both robustness and sensitivity; The parameter vector to be learned according to gradient descent or its variants (such as the adaptive learning rate algorithm) Perform iterative updates and obtain the updated parameters of the short-term model after iterative convergence , forming a new short-term prediction or judgment baseline, where:

[0028] Where: is the learning rate; Indicates Parameter values ​​for the next generation; is the gradient of the log-hyperbolic cost function with respect to the parameters; When in use, the short-term model is corrected in time to adapt it to temporary or sudden changes in load characteristics, reduce the cumulative error transmission to subsequent steps, and through coordination with the long-term update of step 202, a dual control of short-term mutations and long-term trends is formed, allowing the model to more effectively distinguish individual anomalies from overall behavior drift in subsequent group anomaly judgments.

[0029] Step 202: Long-term model periodic maintenance Using longer time series data to test long-term models Conduct periodic training to cope with seasonal and annual changes in electricity consumption habits of a wide range of users; including: Updated short-term model It can provide a reference range for recent normal patterns. If you find a suspected abnormality in the list If some of the loads are verified to be normal loads, they can be used as new normal power consumption pattern samples; and They represent the feature vectors and corresponding load values ​​in a longer period in the past (such as quarter, half year, year), which are used for long-term model training; let the training set size be (It can cover a longer time span), remove the confirmed real abnormal records, and only keep the normal and confirmed transition load data. represents the output function of the long-term model, is the parameter vector to be learned; a periodic smoothing regularization term is introduced, and the following periodic smoothing log-hyperbolic cost is defined as the training target of the long-term model:

[0030] Where: is the regularization coefficient for the long-term model; represents the periodic smoothing regularization term, and an appropriate design can be selected according to business priors such as seasonal periodicity (for example, restricting the parameter differences of the model within the same season segment should not be too large) to enhance the capture of long-term trends; At appropriate periodic nodes (such as monthly, quarterly) or trigger conditions (such as the continuous occurrence of multiple potential pattern change signals), perform batch or mini-batch optimization on the periodic smoothing logarithmic hyperbolic cost :

[0031] where is the learning rate of the long-term model, which can be dynamically adjusted according to the training scale and data distribution; When in use, it can capture the trend of behavior changes on a larger time scale, thus helping subsequent steps quickly distinguish between general seasonal changes and individual abnormal diffusions; when the short-term model has difficulty in accurately distinguishing in the face of periodic load changes, the long-term model can provide a more macroscopic benchmark, providing a relatively stable reference for subsequent group anomaly determination. After receiving the potential pattern changes and suspected abnormal data from step one, perform incremental updates on the short-term model to ensure that the model maintains a high sensitivity to sudden behaviors on a short time scale.

[0032] Step 3. When the updated short-term model and the long-term model receive new load records, perform group anomaly determination and multi-time scale fusion recognition, and identify large-scale electricity consumption pattern transfers or individual anomalies by synthesizing multi-scale anomaly scores and comparing with the anomaly threshold , and at the same time monitor the group deviation rate to output a list of key anomalies and normal anomalies; The content of step 3 includes the following: Step 301. Calculate multi-scale anomaly scores For the user load data from the current time period, apply the short-term model and the long-term model respectively to calculate the multi-scale anomaly score for each record; Let represent the electricity consumption feature vector observed at time (or discrete batch ), and this electricity consumption feature vector covers multi-dimensional load features. Let represent the actual load value or monitored value corresponding to time ; the short-term model and the long-term model received from step two are respectively denoted as With ; To characterize the multi-scale anomaly degree at the moment Define the short-term error vector and the long-term error vector as follows: Short-term error vector: , where Is the error metric under the updated short-term model , and the second component being zero indicates a significant deviation only in the short-term error dimension; Long-term error vector: , where Is the error metric under the updated long-term model , and the first component being zero indicates a significant deviation only in the long-term error dimension; Let Be a 2×2 symmetric positive definite matrix, which is used to reflect the weights and coupling relationships of different directions or components in the vector norm and can be written as:

[0033] It can express the mutual influence between the short-term error and the long-term error to a certain extent (if Indicates that there is a coupling term between the two), and the final multi-scale anomaly score Is defined as:

[0034] In the formula: Is the fusion variable, which gradually changes from 0 to 1 during the integration process and continuously linearly mixes the long-term error vector and the short-term error vector; Represents the weighted two-norm based on ; Is the exponential amplification factor; Set the anomaly threshold . For each record, if the multi-scale anomaly score Anomaly threshold , mark it as a potential anomaly record; if the multi-scale anomaly score Is within the preset gray area (slightly higher or slightly lower than ), then it can be rechecked later; this judgment result will be combined with the population analysis in step 302 to finally determine whether the record enters the key anomaly list or the regular anomaly list. Thus, the sensitivity of the short-term model and the stability of the long-term model can be effectively combined to lay a high-precision foundation for the subsequent population-level judgment. The model output is directly converted into a unified multi-scale anomaly score , which is convenient for batch processing and collaborative discrimination in the subsequent steps.

[0035] Step 302, Population Correlation Analysis and Anomaly Grading For the obtained multi-scale anomaly scores further analyze the collective transfer or large-scale anomalies of electricity consumption patterns at the group level, and output a list of key anomalies and a list of regular anomalies; Let be the set of users to be analyzed currently, and each user has corresponding multi-scale anomaly scores at time ; for the record of each user , if the multi-scale anomaly score is higher than the anomaly threshold and the long-term and short-term errors are both high, it is pre-included in the potential individual anomaly list, otherwise it is regarded as a record at the normal level or in the gray area; To identify whether a large number of users show a trend of increasing or decreasing load simultaneously, the following group deviation rate is defined: :

[0036] where is an indicator function, which takes 1 when and 0 otherwise; If the group deviation rate exceeds the preset group threshold (such as 0.3 or 0.5, set according to business requirements), it means that at the current time a significant proportion of users are in a state of high multi-scale anomaly scores. At this time, it may not be an anomaly, but a sign of overall load transfer or seasonal change. These corresponding records are downgraded to regular anomalies or listed separately as group mode transfer labels to avoid false alarms of a large number of individual cases; When a certain record meets both the criteria of high multi-scale anomaly scores and the group deviation rate corresponding to the time when it is located is not high, it means that this is an individual deviation and is more likely to be a real anomaly, and it is listed in the key anomaly list; If the group deviation rate is high and the deviation of this record in the short-term model or long-term model is also large ( and are both high), then according to the specific business logic, it can be retained in the key anomalies or further distinguished from the group mode transfer; For gray area records ( near the threshold) and at the same time the group deviation rate is in a relatively high range, it can be preferentially determined as a regular anomaly or a group change and transferred to the next step for further manual verification.

[0037] When in use, by introducing the group deviation rate , it can collectively identify large - scale same - direction offsets, avoiding misjudging the load increase caused by overall seasonal changes or social events as a large number of anomalies; combining the multi - scale anomaly scores with the group threshold can achieve differential processing of individual sudden deviations and collective behavior changes, significantly reducing large - scale group false alarms and saving system inspection resources. This helps to accurately select key anomalies, and thus be more flexible and efficient when dealing with load changes of different scales and types. Separate the real anomaly cases into a list of key anomalies, and classify the large - scale common changes into a list of regular anomalies or group pattern transfer labels, realizing more refined anomaly grading and discrimination.

[0038] Step 4: After receiving the list of key anomalies and regular anomalies, through human - machine collaborative feedback and continuous optimization iteration, label information is added to the samples , and the drift judgment threshold is corrected in real - time using the threshold correction value , the multi - scale anomaly threshold and other judgment criteria, and the confirmed normal or abnormal samples are respectively injected into the incremental training pipelines of short - term and long - term models to complete closed - loop self - learning; The content of the above - mentioned Step 4 includes the following: Step 401: Manual review and interactive annotation Review the generated list of key anomalies and regular anomalies to accurately distinguish real anomalies, group transitions, or individual non - malicious behaviors, etc., and write the final conclusion into the system annotation library in a standardized manner; Obtain the list of key anomalies and the list of regular anomalies , and conduct manual or expert - system verification on each record or each user time period (using the time as the index), and output the annotation information . All annotation information will be stored in the annotation library ; Let {Cd, Fm, Un} represent the manual confirmation label for the record, which is a real anomaly, false alarm, or temporarily undetermined (requiring a longer observation period). Send each record in the list of key anomalies to the review end, and judge its nature in combination with actual on - site investigation information (such as load - side equipment inspection, user feedback): If it is confirmed that there is an abnormal behavior (such as electricity theft, line failure), then Cd; if it is determined to be a user behavior change or measurement error, then Fm; if it cannot be determined yet, temporarily assign Un. For the list of regular anomalies Records with relatively minor medium errors or obvious group characteristics can be processed in batches and marked as false alarms or real anomalies; When in use, visualizing and playing back the load curve and the model prediction curve can help operation and maintenance personnel intuitively compare abnormal sections and quickly give annotations, aggregate similar events in the conventional anomaly list into batch annotation tasks, greatly improve the annotation efficiency and ensure consistency. Through the multi-label mechanism, the operation and maintenance costs brought by false alarms can be significantly reduced, ensuring that the confirmed anomalies are more authentic.

[0039] Step 402, Threshold Adaptive Adjustment and Model Iterative Feedback Utilize the annotation information To correct the online drift determination threshold, multi-scale anomaly threshold, etc., and send the confirmed normal or abnormal samples back to the model training pipeline in the second step to gradually improve the discrimination ability of the short-term model and the long-term model; The annotation library obtained from step 401 , which contains the final annotations corresponding to each record at time (); the drift determination threshold and the multi-scale anomaly determination threshold may be updated momentumally or incrementally based on the new annotation information; if the short-term model and the long-term model find newly emerging real anomalies or non-anomaly samples, the model parameters can be corrected through the incremental learning mechanism; to dynamically adjust the drift determination threshold and the multi-scale anomaly threshold according to the continuously emerging annotation information, define the following vector form: Let denote the annotation statistical vector at discrete or continuous time (which can take to represent the iteration batch, or take to represent the actual time), for example: ; where; ; denotes the number of samples or records confirmed as real anomalies within the time period ; denotes the number of samples or records determined as false alarms within the time period ; denotes the number of samples or records that cannot be determined temporarily and are not suitable for inclusion in model training within the time period ; these three components are independent of each other and come from the manual or expert verification results in step 401; Let be a weighted conversion matrix, designed to adjust the influence of various types of annotations during threshold correction and can reflect the interaction between different types of annotations, for example:

[0040] Among them, represents the coupling coefficient of the -th row (corresponding to a certain correction component) to the -th column (corresponding to a certain annotation type), which can be positive or negative; the specific value can be learned from historical data or set by business prior experience; Let be a symmetric positive definite matrix of

[0041] which is used to assign different weights to the outputs of different dimensions in the norm calculation, and its form is: can enhance the sensitivity to certain annotation coupling terms (such as the importance of confirmed anomalies for threshold correction), and finally define the threshold correction value as the weighted norm integral of the annotation vector in the

[0042] interval, with an additional exponential amplification mechanism: is the weighted two-norm based on the matrix ; is the exponential amplification coefficient, when increases, the annotation information volume is relatively high (such as a large number of false alarms or a large number of confirmed anomalies) will be significantly amplified; is the time decay weight, which gives a greater impact to the annotations closer to the current time , and the annotations in the earlier stage gradually decay; represents the integral / summation of continuous or discrete time steps (which can be implemented by numerical discretization).

[0043] When the threshold correction value is continuously large for a period of time, it indicates that there are a large number of false alarms or new anomalies in the system that are not covered by the threshold. At this time, appropriately increase or decrease the drift determination threshold , the multi-scale anomaly threshold : if the threshold correction value remains stable or low, it indicates that the existing threshold can better balance false alarms and missed detections, and the current settings can be maintained.

[0044] Samples that are marked as false alarms Fm but were once judged as anomalies by short-term or long-term models are re-incorporated into the normal training set to reduce the misjudgment probability of the model for similar patterns; true anomaly Cd samples are incorporated into the anomaly training set to help the model strengthen the capture of true anomaly features; uncertain Un samples can be temporarily stored in the candidate set and merged after further verification results are obtained later to avoid introducing interfering data.

[0045] During use, through the threshold correction value continuously track the annotation situation, quantify whether the recognition threshold is too high or too low, and automatically guide the threshold adaptive adjustment. Separate training pipelines are set for the data confirmed as normal and abnormal, significantly enhancing the discriminative ability of the model for different categories of data and stagewise fusion update.

[0046] Dynamically correct the drift determination threshold in step one and the anomaly threshold in step three, enabling the system to promptly respond to offsets caused by changes in user behavior or environmental factors. Through continuous model iteration, avoid system aging and maintain high detection accuracy; cooperate with step 401: the confirmed anomalies or false alarms are quickly reflected in the system threshold and model parameters, shortening the feedback closed-loop duration.

[0047] Furthermore, the annotation information ensures the accuracy of anomaly determination, and at the same time provides high-quality evidence for model parameter update and threshold correction. Threshold update and model iteration can further reduce the false alarm rate and missed alarm rate, making the next round of detection more sensitive and reliable. The multi-class label (confirmed, false alarm, uncertain) strategy enhances the adaptability of the system, can defer decisions when there is a lack of clear information, and avoid model misalignment caused by noise; through the above iterative process, the solution can continuously optimize the detection baseline and model when facing the continuous evolution of user electricity consumption behavior or environmental factor impacts, and finally significantly reduce the false alarms and missed alarms caused by concept drift, achieving a truly closed-loop self-learning anomaly detection system.

[0048] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraint conditions of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0049] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described system, device, and unit can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0050] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only for some logical function divisions. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0051] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0052] As described above, this is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for detecting abnormal power consumption based on big data, characterized in that: including Step 1: When there is a significant deviation between the load distribution and the historical prediction result, trigger online concept drift detection. Use the sliding window and threshold monitoring to collaboratively calculate the logarithmic hyperbolic error to identify the sudden change in the distribution, mark the potential pattern change, and generate a list of suspected anomalies. Step 2: After receiving the potential pattern change mark and the list of suspected anomalies, trigger the model adaptive update. Use incremental training to correct the short-term model and periodic training for the long-term model, generate a new prediction baseline, and incorporate the latest normal load pattern. Step 3: When obtaining new load records, perform population anomaly determination and multi-scale fusion recognition. Calculate the error metrics using the long-term model and the short-term model respectively, aggregate them to generate a multi-scale anomaly score, and then generate a list of key anomalies and regular anomalies by comparing with the multi-scale anomaly threshold. Step 4: After receiving the list of key anomalies and regular anomalies, trigger human-machine collaborative feedback and continuous optimization iteration. Use the manual annotation results to correct the drift determination threshold and the multi-scale anomaly threshold, and inject the annotated samples into the incremental training pipeline to complete the closed-loop self-learning.

2. The big data-based power and electricity anomaly detection method according to claim 1, characterized in that: Step 1 includes step 101: Collect the load distribution and reference data within the detection window, and analyze the logarithmic stretching deviation at each moment. Introduce a weighted accumulation strategy to obtain the comprehensive deviation value at the current moment. When and only when the comprehensive deviation value is greater than the drift determination threshold, mark the data in the corresponding period as a potential pattern change. Otherwise, only save the regular anomaly records to the regular anomaly list.

3. The big data-based power and electricity anomaly detection method according to claim 1, characterized in that: After step 101, there is step 102: For each load record in the load record set, compare it with the existing short-term or long-term prediction values to form a preliminary anomaly determination according to the preset business rules. Mark the records with significant difference as suspected anomaly labels, package all suspected anomaly records into a list of suspected anomalies, and output the possible drift period mark.

4. The big data-based power and electricity anomaly detection method according to claim 3, characterized in that: Step 2 includes step 201: Collect the data records that intersect with the pattern change mark and the list of suspected anomalies in the recent multiple windows to form a short-term training set. After using the logarithmic hyperbolic cost as the objective function of incremental learning, iterate and update the parameter vector to be learned according to gradient descent or its variant. After the iteration converges, obtain the updated parameters of the short-term model to form a new short-term prediction or discrimination baseline.

5. The big data-based power and electricity anomaly detection method according to claim 4, characterized in that: After step 201, there is step 202: Set a training set that only retains normal and confirmed transitional load data, define the periodic smoothed logarithmic hyperbolic cost as the training objective of the long-term model, and perform batch or mini-batch optimization on the periodic smoothed logarithmic hyperbolic cost.

6. The big data-based power and electricity anomaly detection method according to claim 1, characterized in that: Step 3 includes step 301, which respectively applies a short-term model and a long-term model to the power load data from the current period to calculate multi-scale anomaly scores for each record; For each record, if the multi-scale anomaly score is higher than the anomaly threshold, it is marked as a potential anomaly record; if the multi-scale anomaly score is within the preset gray area range, it can be rechecked later.

7. The big data-based power quantity anomaly detection method according to claim 6, characterized in that: After step 301, there is step 302. For the records of each user, if the multi-scale anomaly score is higher than the anomaly threshold and the long-term and short-term errors are both higher than expected, it is pre-included in the potential individual anomaly list, otherwise it is regarded as a record at the normal level or in the gray area; the group deviation rate is constructed to identify whether a large number of users show a trend of increasing or decreasing load at the same time, so as to distinguish between overall pattern transfer and individual anomalies, and generate a list of key anomalies and normal anomalies.

8. The big data-based power quantity anomaly detection method according to claim 7, characterized in that: Step 4 includes step 401, which conducts a review on the generated list of key anomalies and the list of normal anomalies, outputs annotation information and stores it in the annotation library; For the records in the list of normal anomalies with minor errors or obvious group characteristics, batch processing is performed and they are marked as false alarms or real anomalies.

9. The big data-based power quantity anomaly detection method according to claim 8, characterized in that: After step 401 is executed, if the short-term model and the long-term model find newly emerging real anomaly or non-anomaly samples, the drift determination threshold and the multi-scale anomaly threshold are updated in a momentum or incremental manner according to the new annotation information, and the confirmed normal or anomaly samples are sent back to the model training pipeline in the second step.

10. The big data-based power quantity anomaly detection method according to claim 9, characterized in that: When the threshold correction value constructed through integral and exponential amplification operations continues to be greater than expected for a period of time, the values of the drift determination threshold and the multi-scale anomaly threshold are increased or decreased. If the threshold correction value remains stable or lower than expected, the current settings are maintained; The samples marked as false alarms but previously judged as anomalies by the short-term or long-term model are re-included in the normal training set, the real anomaly samples are included in the anomaly training set, and the uncertain samples are temporarily stored in the candidate set.

Images