Power anomaly detection method based on big data

Abstract

The present invention discloses a method for detecting anomalies of electric power and electricity based on big data, which relates to the technical field of anomaly detection of electric power and electricity. Aiming at the problem of high false alarms and missed alarms caused by dynamic changes in the electricity consumption behavior of power users, the method provides online concept drift detection, model adaptive updating, group anomaly determination and multi-scale fusion recognition. Potential pattern changes are discovered through sliding windows and thresholds, and a list of suspected anomalies is generated. Short-term models are corrected using incremental training and long-term models are periodically updated. Subsequently, multi-scale error metrics are fused into multi-scale anomaly scores, and key anomalies and routine anomalies are determined in combination with group deviation rates. Finally, adaptive adjustments are made through threshold correction indicators and the manual annotation results are fed back to the model library, so as to achieve accurate distinction and continuous learning between normal behavior changes and real anomalies. Through this solution, the false alarm and missed alarm rates can be significantly reduced, and the reliability and refinement of power dispatching and monitoring can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power and electricity anomaly detection, and in particular to a power and electricity anomaly detection method based on big data. Background Art

[0002] In the modern power industry's electricity consumption big data landscape, load behavior across various user types is highly dynamic and diverse. For residential users, for example, load curves can fluctuate significantly over relatively short periods due to factors such as changes in household structure, the addition of new appliances (such as electric vehicles and smart air conditioners), and holiday and seasonal changes. Power consumption can surge or plummet at night and during peak hours. Industrial and commercial users also face dynamic electricity demand. When production lines expand or decrease in capacity, overall energy consumption can also experience sharp jumps in the short term. Furthermore, in localized areas or during special periods (such as during high temperatures, severe cold weather, or public health events), collective electricity consumption can experience overall increases or decreases, resulting in widespread, multi-timescale load pattern shifts. With the continuous upgrade of power metering and dispatching networks, this massive amount of electricity consumption data is being collected in real time and applied to operational monitoring, energy efficiency management, and intelligent dispatching. However, since user load behavior is not static, if normal behavior drift and true abnormal events cannot be distinguished in a timely manner, traditional anomaly detection methods based on fixed thresholds or historical patterns are prone to a large number of misjudgments in actual operation and maintenance, resulting in management difficulties such as waste of scheduling resources and delayed troubleshooting.

[0003] To address these dynamic load characteristics, existing power anomaly detection systems often use fixed model parameters or empirical thresholds derived from earlier training. These systems lack the ability to adaptively learn from concept drift, resulting in a large number of normal changes in real-world scenarios being falsely reported as anomalies. Furthermore, they may miss reports of progressive power theft or equipment failures. Traditional methods rely primarily on initial historical data to establish "normal" electricity usage patterns, assuming a relatively stable future distribution. However, they ignore sudden changes in distribution caused by factors such as the addition of new power equipment and seasonal changes. Even some systems employ regular manual updates or retrain models during annual inspections, but these systems often lack the ability to respond to frequent changes in user behavior due to lags. When the load curve experiences significant increases or decreases within a short period, static thresholds cannot accurately capture the difference between "normal changes" and "true anomalies," leading to an uncontrollable surge in false alarms. Furthermore, abnormal samples are often scarce and diverse, and supervised learning faces the challenge of uneven distribution, resulting in insensitivity to emerging anomaly types.

[0004] In summary, this lack of adaptive response to concept drift and difficulty in distinguishing true anomalies from normal pattern changes have become the main bottleneck for intelligent monitoring and efficient operation and maintenance of power grids. Summary of the Invention

[0005] (1) Technical problems solved

[0006] In response to the shortcomings of the existing technology, the present invention provides a method for detecting power anomalies based on big data, which discovers potential pattern changes through sliding windows and thresholds and generates a list of suspected anomalies; uses incremental training to correct short-term models and periodically updates long-term models; then fuses them into multi-scale anomaly scores based on multi-scale error metrics, and combines group deviation rates to determine key anomalies and routine anomalies; finally, adaptively adjusts through threshold correction indicators and returns the manual annotation results to the model library, thereby achieving accurate distinction and continuous learning between normal behavior changes and real anomalies, which can significantly reduce false alarm and missed alarm rates, and improve the reliability and refinement of power dispatching and monitoring, thereby solving the technical problems recorded in the background technology.

[0007] (2) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for detecting power anomalies based on big data, comprising:

[0009] When the load distribution deviates significantly from historical forecast results, online concept drift detection is triggered. A sliding window and threshold monitoring are used to calculate the logarithmic hyperbolic error to identify sudden changes in the distribution, mark potential pattern changes, and generate a list of suspected anomalies.

[0010] After receiving the potential pattern change marker and the list of suspected anomalies, the model adaptive update is triggered. Incremental training is used to correct the short-term model and periodically train the long-term model to generate a new prediction baseline and incorporate the latest normal load pattern.

[0011] When a new load record is obtained, group anomaly determination and multi-scale fusion recognition are performed. Error metrics are calculated using long-term and short-term models respectively, and then aggregated to generate multi-scale anomaly scores. Then, a list of key anomalies and general anomalies is generated by comparing the multi-scale anomaly threshold.

[0012] After receiving the list of key anomalies and common anomalies, human-machine collaborative feedback and continuous optimization iteration are triggered. The drift judgment threshold and multi-scale anomaly threshold are corrected using the manual labeling results, and the labeled samples are injected into the incremental training pipeline to complete closed-loop self-learning.

[0013] Preferably, the load distribution and reference data within the detection window are collected, and the logarithmic tensile deviation at each moment is analyzed;

[0014] A weighted accumulation strategy is introduced to obtain the comprehensive deviation value at the current moment. If and only if the comprehensive deviation value is greater than the drift judgment threshold, the data of the corresponding period will be marked as a potential pattern change. Otherwise, only regular anomaly records will be saved in the regular anomaly list.

[0015] Preferably, each load record in the load record set is compared with an existing short-term or long-term forecast value, and a preliminary abnormality determination is formed according to pre-set business rules;

[0016] The records with significant differences are labeled as suspected anomalies, all suspected anomaly records are packaged into a suspected anomaly list, and the possible drift period mark is output.

[0017] Preferably, data records that intersect with pattern change markers and suspected anomaly lists in recent multiple windows are collected to form a short-term training set. After using the logarithmic hyperbolic cost as the objective function of incremental learning, the parameter vector to be learned is iteratively updated according to gradient descent or its variants. After iterative convergence, the updated parameters of the short-term model are obtained to form a new short-term prediction or discrimination baseline.

[0018] Preferably, a training set is set to retain only normal and confirmed transition load data, and a periodic smoothed logarithmic hyperbolic cost is defined as a training target for the long-term model, and the periodic smoothed logarithmic hyperbolic cost is optimized in batches or small batches.

[0019] Preferably, for the power load data from the current period, a short-term model and a long-term model are applied respectively to calculate a multi-scale anomaly score for each record;

[0020] For each record, if the multi-scale anomaly score is higher than the anomaly threshold, it is marked as a potential anomaly record; if the multi-scale anomaly score is within the preset gray zone, it can be re-checked later.

[0021] Preferably, for each user's record, if the multi-scale anomaly score is higher than the anomaly threshold and the long-term and short-term errors are high at the same time, it will be included in the potential individual anomaly list in advance, otherwise it will be regarded as a normal level or gray area record;

[0022] Construct a group deviation rate to identify whether a large range of users have a load increase or decrease trend at the same time, so as to distinguish overall pattern shifts from individual anomalies and generate a list of key anomalies and regular anomalies.

[0023] Preferably, the generated key anomaly list and routine anomaly list are reviewed, and the annotation information is output and stored in the annotation library; records with slight errors or obvious group characteristics in the routine anomaly list are batch processed and marked as false alarms or real anomalies.

[0024] Preferably, if the short-term model and the long-term model discover newly emerged real abnormal or non-abnormal samples, they perform momentum or incremental updates on the drift judgment threshold and the multi-scale abnormal threshold based on the new annotation information, and send the confirmed normal or abnormal samples back to the second step of the model training pipeline.

[0025] Preferably, when the threshold correction value constructed by the integration and exponential amplification operation is continuously greater than expected for a period of time, the values ​​of the drift determination threshold and the multi-scale anomaly threshold are increased or decreased; if the threshold correction value remains stable or lower than expected, the current setting is maintained;

[0026] Samples that are marked as false positives but have been judged as abnormal by short-term or long-term models are re-included in the normal training set, real abnormal samples are included in the abnormal training set, and uncertain samples are temporarily stored in the candidate set.

[0027] (3) Beneficial effects

[0028] The present invention provides a method for detecting power anomalies based on big data, which has the following beneficial effects:

[0029] Using drift judgment threshold Combining sliding windows with real-time monitoring, relevant data will be marked as potential pattern changes when significant deviations occur, and a list of suspected anomalies will be output to ensure timely capture of sudden and trend changes.

[0030] For potential drift data, use incremental learning or sliding window training to train short-term models. With the long-term model Adaptive updates are performed to incorporate new load patterns that are confirmed to be normal into the normal baseline, greatly improving the robustness in dealing with seasonal changes, new high-power equipment, etc.

[0031] Exploiting short-term errors and long-term error Constructing multi-scale anomaly scores , and combined with the analysis of group deviation rate, according to the abnormal threshold The abnormality level is subdivided: if most users show a load increase in the same direction at the same time, it is judged as a regular abnormality or a group pattern change. Otherwise, individual deviations are included in the key abnormality list, significantly reducing concentrated false alarms.

[0032] After obtaining key anomalies and regular anomalies, the label results are produced through review and interactive annotation, and the threshold correction value is used. Feedback to drift judgment threshold , multi-scale anomaly threshold Dynamic tuning and providing new positive and negative samples for short-term and long-term models. The closed-loop self-learning mechanism can continuously accumulate abnormal patterns in real scenarios, suppress the cumulative effect of false positives and missed negatives, and maintain sufficient sensitivity to a small number of high-risk anomalies. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The figure is a flow chart of the method for detecting abnormal power and electricity quantity based on big data of the present invention. DETAILED DESCRIPTION

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0035] See also Figure 1 The present invention provides a method for detecting power anomalies based on big data, comprising:

[0036] Step 1: When a significant difference is detected between the load distribution of the current period and the prediction results of the historical model, online concept drift detection and preliminary screening are carried out for real-time or quasi-real-time data. Adopting sliding window and threshold monitoring, and judging statistical deviations, it then implements segmented alarm screening based on the degree of threshold exceedance, outputs potential pattern changes, and annotates a list of suspected anomalies, which is then stored in the monitoring database;

[0037] The step 1 includes the following:

[0038] Step 101: Weighted distribution deviation measurement

[0039] A custom weighted distribution deviation measurement function is introduced to determine whether concept drift occurs by monitoring the load distribution within the window in real-time or quasi-real-time. The specific technical logic is as follows:

[0040] make Indicates time (or discrete batch ) contains multiple components representing the characteristics of electricity consumption; Representation and time The corresponding reference distribution vector can be given by a pre-trained history model or prediction module;

[0041] Set the sliding window length to , used to collect the most recent Distribution deviation information of each period; Select the attenuation factor , used to give decreasing weight to the deviation contribution of earlier periods in the accumulation process; For each moment in time, the logarithmic stretch deviation is calculated as follows:

[0042]

[0043] in: is the load distribution vector The number of dimensions (i.e., the number of components) is the deviation stretching coefficient, which is greater than 0 and is used to amplify or suppress the sensitivity of distribution differences; To prevent a tiny constant with a zero denominator; and Respectively The load characteristics at time The actual value and reference value of

[0044] In order to make the detection more sensitive to recent deviations, a weighted accumulation strategy is introduced to obtain the current moment The comprehensive deviation value of :

[0045]

[0046] Set a drift determination threshold , if and only if the comprehensive deviation value Greater than the drift judgment threshold When, determine at the moment Significant concept drift may have occurred within the corresponding window segment, triggering subsequent preliminary screening;

[0047] When used, it can promptly detect obvious shifts in user load distribution over time, avoiding misjudgment of subsequent processes due to sudden fluctuations in a single scenario (such as a user's high-power equipment being turned on), and provides highly targeted information on possible drift periods, reducing repeated calculations of indifferent intervals and improving overall detection efficiency.

[0048] Step 102: Preliminary abnormality screening and data labeling

[0049] Based on the obtained possible drift period determination results, the load data of the corresponding period is screened for preliminary anomalies and a list of suspected anomalies is output. If the drift threshold is not reached, only regular anomaly records are retained, as follows:

[0050] Get the possible drift period mark: when Time, corresponding to the moment and the segments within its sliding window, automatically marking the data in that period as potential pattern change markers ; Let denote the set of all load records in the segment;

[0051] like , it is considered that no obvious concept drift is detected, and only regular exception records are saved in the regular exception list; for the load record set Each load record in the system is compared with the existing short-term or long-term forecast values, and a preliminary abnormality judgment is made according to the pre-set business rules (such as the upper and lower limits of basic power, the rationality of power consumption time, etc.); the record with significant difference (exceeding the set business rule threshold) is marked with a suspected abnormality label, and is associated with the corresponding possible drift period for storage, and all suspected abnormal records are packaged into a suspected abnormality list , and output possible drift period mark;

[0052] When using, use the load record collection Achieve targeted screening, avoid comparing massive amounts of data one by one, improve processing performance, and list suspected anomalies Closely linked to the possible drift period, this provides valuable clues for subsequent model updates, enabling differentiated treatment of new and old baselines. This significantly reduces ineffective computations in the indifferent intervals during subsequent steps. By delineating the possible drift period, resources are centralized and the initial screening results are structured for output, ensuring deeper model adaptation and feature correction for suspected anomalies in the next step, significantly improving the accuracy and efficiency of subsequent processing.

[0053] Step 2: After receiving the potential pattern change mark and the suspected anomaly list, perform model adaptive update and multi-time scale maintenance, targeting short-term model parameters Use incremental training to handle recent sudden changes and adjust long-term model parameters Use periodic training to learn seasonal trends, incorporate them to confirm normal behavior or retain abnormal feature labels, and output new forecast baselines through updated short-term and long-term models;

[0054] The second step includes the following:

[0055] Step 201: Short-term model incremental update

[0056] Flagged for potential schema changes and a list of suspected anomalies The core of the incremental update of short-term models is to quickly correct model parameters with the help of recent load data, so that the model can adapt to possible sudden changes in user electricity consumption in a timely manner; mode change marker It represents the concept drift period detected in step 1. The load data in these periods have a greater probability of representing a new behavior pattern; Suspected abnormal data within the same or adjacent time periods, if part of it is confirmed to be normal through subsequent processes, can be regarded as a sample of short-term pattern changes;

[0057] make Indicates time (or discrete batch ) of the load characteristic vector (including multi-dimensional features, such as the power of different electrical equipment, time period, etc.); let Indicates at time The actual load (or corresponding monitoring value) of is used as the learning target of the short-term model; represents the output function of the short-term model, is the parameter vector to be learned;

[0058] Select window size , collect recent Within a moment, and mode change mark and a list of suspected anomalies There are data records with intersections, forming a short-term training set , define the following log-hyperbolic cost as the objective function of incremental learning:

[0059]

[0060] Where: It is a time decay factor, which makes the data closer to the current one have a higher weight in incremental optimization; It is a hyperbolic cosine function that can maintain smoothness when the error is small and provide stronger penalties when the error is large, thus taking into account both robustness and sensitivity;

[0061] The parameter vector to be learned according to gradient descent or its variants (such as adaptive learning rate algorithm) Perform iterative updates and obtain the updated parameters of the short-term model after iterative convergence , forming a new short-term prediction or discrimination baseline, where:

[0062]

[0063] Where: is the learning rate; Indicates the Parameter values ​​of the next generation; is the gradient of the log-hyperbolic cost function with respect to the parameters;

[0064] When in use, the short-term model is corrected in a timely manner to adapt it to temporary or sudden changes in load characteristics, reducing the cumulative error transmission to subsequent steps. By coordinating with the long-term update of step 202, a dual control of short-term mutations and long-term trends is formed, allowing the model to more effectively distinguish individual anomalies from overall behavior drift in subsequent group anomaly judgments.

[0065] Step 202: Long-term model periodic maintenance

[0066] Long-term models using longer time series data Conduct periodic training to cope with seasonal and annual changes in user electricity usage habits.

[0067] Updated short-term model It can provide a reference range for the recent normal mode. If it is in the suspected abnormal list If some of the loads are verified to be normal loads, they can be used as new normal power consumption pattern samples; and Represent the feature vectors and corresponding load values ​​in a longer period (such as quarter, half year, year) in the past, which are used for long-term model training; let the training set size be (can cover a longer time span), remove the confirmed real abnormal records, and only keep the normal and confirmed transition load data. represents the output function of the long-term model, is the parameter vector to be learned; a periodic smoothing regularization term is introduced, and the following periodic smoothing log-hyperbolic cost is defined as the training target of the long-term model:

[0068]

[0069] Where: is the regularization coefficient of the long-term model; represents the periodic smoothing regularization term. An appropriate design can be selected based on business priors such as seasonal periodicity (for example, the parameter differences of the constraint model within the same seasonal period should not be too large) to enhance the capture of long-term trends.

[0070] At appropriate periodic nodes (such as monthly, quarterly) or trigger conditions (such as the continuous occurrence of multiple potential pattern change signals), the periodic logarithmic hyperbolic cost is smoothed. Perform batch or mini-batch optimization:

[0071]

[0072] in The learning rate of the long-term model can be dynamically adjusted according to the training scale and data distribution;

[0073] When used, it can capture behavioral change trends on a larger time scale, thereby helping subsequent steps to quickly distinguish whether it is a general seasonal shift or individual abnormal diffusion; when the short-term model is difficult to accurately distinguish when facing periodic load changes, the long-term model can provide a more macro benchmark and provide a more stable reference for subsequent group anomaly judgment. After receiving potential pattern changes and suspected abnormal data from step one, the short-term model is incrementally updated to ensure that the model maintains high sensitivity to sudden behaviors on a short time scale.

[0074] Step 3: When the updated short-term model With long-term model After receiving the new load record, group anomaly determination and multi-time scale fusion recognition are performed, and the multi-scale anomaly score is synthesized. and with the anomaly threshold Compare and simultaneously monitor group deviation rates to identify large-scale power usage pattern shifts or individual anomalies, and output a list of key anomalies and regular anomalies;

[0075] The step three includes the following:

[0076] Step 301: Multi-scale anomaly score calculation

[0077] Apply short-term models to user load data from the current period and long-term models , calculate a multi-scale anomaly score for each record ;

[0078] make Indicates at time (or discrete batch ) The power consumption feature vector observed by Indicates time The corresponding actual load value or monitoring value; the short-term model received from step 2 With long-term model Denoted as and ;

[0079] To depict the moment The multi-scale anomaly degree under the condition of θ is defined as follows:

[0080] Short-term error vector: ,in Updated short-term model The error metric under , the second component is zero, indicating that there is a significant deviation only in the short-term error dimension;

[0081] Long-term error vector: ,in Updated long-term model The error metric under , the first component is zero, indicating that there is a significant deviation only in the long-term error dimension;

[0082] make is a 2×2 symmetric positive definite matrix, which is used to reflect the weight and coupling relationship of different directions or components in the vector norm and can be written as:

[0083]

[0084] It can express the mutual influence between short-term error and long-term error to a certain extent (if Indicates that there is a coupling term between the two), the final multi-scale anomaly score Defined as:

[0085]

[0086] Where: To fuse the variables, the long-term error vector is linearly mixed with the short-term error vector continuously by gradually changing from 0 to 1 during the integration process;

[0087] Indicates based on The weighted two-norm of ; is the exponential amplification factor;

[0088] Setting anomaly thresholds , for each record, if the multi-scale anomaly score Abnormal threshold , marked as potential abnormal records; if the multi-scale anomaly score Within the preset gray area (slightly above or below ) can be reviewed again later; the judgment result will be combined with the group analysis in step 302 to ultimately determine whether the record will be included in the key anomaly list or the regular anomaly list. This effectively combines the sensitivity of the short-term model with the stability of the long-term model, laying a high-precision foundation for subsequent group-level judgments and directly converting the model output into a unified multi-scale anomaly score. , which facilitates batch processing and collaborative identification in subsequent steps.

[0089] Step 302: Group association analysis and abnormality classification

[0090] Multi-scale anomaly scores obtained , further analyze the collective shift or large-scale anomaly of electricity consumption patterns at the group level, and output a list of key anomalies and a list of general anomalies;

[0091] set up is the set of users to be analyzed, each user At the moment There are corresponding multi-scale anomaly scores ; For each user If the multi-scale anomaly score Abnormal threshold If both the long-term and short-term errors are high, they will be included in the potential individual abnormality list in advance, otherwise they will be regarded as normal level or gray area records;

[0092] In order to identify whether a large range of users have a load increase or decrease trend at the same time, the following group deviation rate is defined: :

[0093]

[0094] in, is the indicator function, when Take 1 when it is, otherwise take 0;

[0095] If the group deviation rate Exceeding the preset group threshold (such as 03 or 05, set according to business needs), it means that at the current time A significant proportion of users have high multi-scale anomaly scores. This may not be anomalies, but rather a sign of overall load shift or seasonal changes. These corresponding records are downgraded to regular anomalies or listed separately as group mode shift labels to avoid false positives for a large number of cases.

[0096] When a record meets both high multi-scale anomaly scores Standard, and at the moment The corresponding group deviation rate If it is not high, it means that this is an individual deviation and is more likely to be a true abnormality and should be included in the list of key abnormalities;

[0097] If the group deviation rate is high and the deviation of this record in the short-term model or the long-term model is also large ( and If both are high), then depending on the specific business logic, it can be retained as a key anomaly or further distinguished from group mode shifts;

[0098] For gray area records ( near threshold) and the population deviation rate If it is in a higher range, it can be prioritized as a routine anomaly or group shift and transferred to the next step for further manual verification.

[0099] When used, by introducing the group deviation rate , can collectively identify large-scale unidirectional excursions, avoiding misjudging the load increase caused by overall seasonal changes or social events as a large number of anomalies; multi-scale anomaly scores and group threshold Combined, differentiated handling of individual sudden deviations and collective behavioral shifts can significantly reduce large-scale group false alarms and save system audit resources. This helps accurately select key anomalies, thereby providing greater flexibility and efficiency in responding to load changes of varying scales and types. By individually refining true anomalies into a list of key anomalies and categorizing large-scale common changes into a list of regular anomalies or group pattern shift labels, more refined anomaly classification and identification can be achieved.

[0100] Step 4: After receiving the list of key anomalies and general anomalies, the samples are annotated with information through human-machine collaborative feedback and continuous optimization iteration. , and use the threshold correction value Real-time correction of drift judgment threshold , multi-scale anomaly threshold The normal or abnormal samples confirmed are injected into the incremental training pipeline of the short-term and long-term models respectively to complete closed-loop self-learning;

[0101] The step 4 includes the following contents:

[0102] Step 401: Manual review and interactive annotation

[0103] Review the generated key anomaly list and regular anomaly list to accurately distinguish between true anomalies, group shifts, or individual non-malicious behaviors, and write the final conclusions into the system annotation library in a standardized manner;

[0104] Get a list of key exceptions With the general exception list , for each record or each user period (take time as an index) for manual or expert system verification and output of annotation information , all annotation information Will be stored in the annotation library middle;

[0105] make {Cd, Fm, Un} represents the manually confirmed label of the record, which is a real anomaly, a false alarm or temporarily uncertain (a longer observation period is required). Each record in the report is sent to the review end, and its nature is determined based on actual on-site investigation information (such as load-side equipment inspection and user feedback):

[0106] If abnormal behavior is confirmed (such as electricity theft, line failure), then Cd; If it is determined to be a user behavior change or measurement error, then Fm; if it cannot be determined yet, temporarily assign a value Un, for general exception list Records with minor errors or obvious group anomalies can be processed in batches and marked as false positives or true anomalies;

[0107] When in use, visual playback of the load curve and model prediction curve can help operation and maintenance personnel intuitively compare abnormal sections and quickly give labels, and aggregate similar events in the regular exception list into batch labeling tasks, greatly improving labeling efficiency while taking into account consistency. Through the multi-label mechanism, the operation and maintenance costs caused by false alarms can be significantly reduced, ensuring that confirmed anomalies are more authentic.

[0108] Step 402: Threshold adaptive adjustment and model iterative feedback

[0109] Using annotation information Correct the online drift judgment threshold and multi-scale anomaly threshold, and send the confirmed normal or abnormal samples back to the second step of the model training pipeline to gradually improve the discrimination ability of the short-term model and the long-term model;

[0110] The annotation library obtained from step 401 , which is included in the moment The final annotations corresponding to the following records ( Drift determination threshold and multi-scale anomaly determination threshold , which may be updated incrementally or incrementally based on new annotation information; if the short-term model and the long-term model find new real abnormal or non-abnormal samples, the model parameters can be corrected through the incremental learning mechanism; in order to dynamically adjust the drift judgment threshold according to the continuously appearing annotation information and multi-scale anomaly threshold Define the following vector form:

[0111] make Indicates whether it is in discrete or continuous time (Desirable Represents an iteration batch, or takes represents the actual time), for example: ;in; Indicates the time period The number of samples or records confirmed as true anomalies; Indicates the time period The number of samples or records judged as false positives; Indicates the time period The number of samples or records that cannot be determined temporarily and are not suitable for inclusion in model training; these three components are independent of each other and come from the manual or expert verification results of step 401;

[0112] make For one The weighted transformation matrix is ​​designed to adjust the influence of various annotations during threshold correction and reflect the interaction between different types of annotations, for example:

[0113]

[0114] in, Indicates the The coupling coefficient of a row (corresponding to a certain correction component) to the jth column (corresponding to a certain annotation type) can be positive or negative; the specific value can be learned from historical data or set based on business prior experience;

[0115] make For one A symmetric positive definite matrix is ​​used to give differentiated weights to outputs of different dimensions in norm calculation, and its form is:

[0116]

[0117] It can enhance the sensitivity of certain annotation coupling items (such as the importance of confirmed anomalies to threshold correction) and ultimately define the threshold correction value For the label vector Weighted norm integration of intervals with an additional exponential amplification mechanism:

[0118]

[0119] Where: Based on the matrix The weighted two-norm of ;

[0120] is the exponential amplification factor, when When it increases, the amount of annotation information is high (such as a large number of false positives or a large number of confirmed anomalies) will be significantly magnified; is the time decay weight, which is closer to the current moment The annotations of the early stages have a greater impact, and the annotations of the earlier stages gradually decay; Represents the integration / summation of continuous or discrete steps in time (can be implemented using numerical discretization).

[0121] When the threshold correction value If the value is continuously too high for a period of time, it indicates that there are a lot of false alarms in the system or new anomalies are not covered by the threshold. In this case, it is appropriate to increase or decrease the drift judgment threshold. , multi-scale anomaly threshold The value of: If the threshold correction value If the threshold remains stable or low, it means that the existing threshold can well balance false positives and false negatives, and the current setting can be maintained.

[0122] Samples marked as false positives Fm but previously identified as abnormal by short-term or long-term models are re-included in the normal training set to reduce the probability of the model misjudging similar patterns. Real abnormal samples Cd are included in the abnormal training set to help the model strengthen its ability to capture real abnormal features. Uncertain samples Un can be temporarily stored in the candidate set and merged after further verification to avoid introducing interference data.

[0123] When used, the value is corrected by the threshold Continuously track the labeling situation, quantify whether the recognition threshold is too high or too low, and automatically guide the threshold to be adaptively adjusted. Set up separate training pipelines for data confirmed as normal and data confirmed as abnormal, significantly improving the model's discriminative power for different categories of data and performing phased fusion updates.

[0124] Dynamically correct the drift judgment threshold in step one and the abnormality threshold in step three so that the system can respond promptly to changes in user behavior or deviations caused by environmental factors. Through continuous model iteration, system aging is avoided and high detection accuracy is maintained. This works in synergy with step 401: confirmed abnormalities or false alarms are quickly reflected in the system thresholds and model parameters, shortening the feedback loop duration.

[0125] Furthermore, the labeled information ensures the accuracy of anomaly judgment and provides high-quality evidence for model parameter updates and threshold corrections. Threshold updates and model iterations can further reduce false alarm and missed alarm rates, making the next round of detection more sensitive and reliable. The multi-category label (confirmation, false alarm, uncertainty) strategy enhances the system's adaptability and can postpone decisions in the absence of clear information to avoid model inaccuracy caused by noise. Through the above iterative process, the solution can continuously adjust the detection baseline and model when facing the continuous evolution of user electricity consumption behavior or the impact of environmental factors, ultimately significantly reducing false alarms and missed alarms caused by concept drift, and achieving a truly closed-loop self-learning anomaly detection system.

[0126] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0127] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only for some logical functions. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0129] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0130] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for detecting power anomalies based on big data, characterized by: include, In step 1, when the load distribution deviates significantly from historical forecast results, online concept drift detection is triggered. A sliding window and threshold monitoring are used to calculate the log-hyperbolic error to identify sudden changes in the distribution, flag potential pattern changes, and generate a list of suspected anomalies. Step 2: After receiving the potential pattern change marker and the list of suspected anomalies, the model adaptive update is triggered. Incremental training is used to correct the short-term model and periodically train the long-term model to generate a new prediction baseline that incorporates the latest normal load pattern. Step 3: When a new load record is obtained, group anomaly determination and multi-scale fusion recognition are performed. Error metrics are calculated using long-term and short-term models respectively, and then aggregated to generate a multi-scale anomaly score. The multi-scale anomaly threshold is then used to generate a list of key anomalies and common anomalies. Step 4: After receiving the list of key anomalies and common anomalies, human-machine collaborative feedback and continuous optimization iteration are triggered. The drift judgment threshold and multi-scale anomaly threshold are corrected using the manual labeling results, and the labeled samples are injected into the incremental training pipeline to complete closed-loop self-learning.

2. The method for detecting power anomalies based on big data according to claim 1, characterized in that: The step 1 includes step 101, collecting load distribution and reference data within the detection window, and analyzing the logarithmic stretching deviation at each moment; A weighted accumulation strategy is introduced to obtain the comprehensive deviation value at the current moment. If and only if the comprehensive deviation value is greater than the drift judgment threshold, the data of the corresponding period will be marked as a potential pattern change. Otherwise, only regular anomaly records will be saved in the regular anomaly list.

3. The method for detecting power anomalies based on big data according to claim 2, characterized in that: After step 101, step 102 is performed to compare the difference between each load record in the load record set and the existing short-term or long-term forecast value, and to form a preliminary abnormality determination according to the pre-set business rules; The records with significant differences are labeled as suspected anomalies, all suspected anomaly records are packaged into a suspected anomaly list, and the possible drift period mark is output.

4. The method for detecting power anomalies based on big data according to claim 3, characterized in that: The step 2 includes step 201, collecting data records that intersect with the pattern change mark and the suspected anomaly list in the most recent multiple windows to form a short-term training set. After using the logarithmic hyperbolic cost as the objective function of incremental learning, the parameter vector to be learned is iteratively updated according to gradient descent or its variants. After iterative convergence, the updated parameters of the short-term model are obtained to form a new short-term prediction or discrimination baseline.

5. The method for detecting power anomalies based on big data according to claim 4, characterized in that: Step 202 is followed by step 201, which sets a training set that only retains normal and confirmed transition load data, defines a periodic smoothed logarithmic hyperbolic cost as a training target for a long-term model, and performs batch or mini-batch optimization on the periodic smoothed logarithmic hyperbolic cost.

6. The method for detecting power anomalies based on big data according to claim 1, characterized in that: The step three includes step 301, applying a short-term model and a long-term model to the power load data from the current period, and calculating a multi-scale anomaly score for each record; For each record, if the multi-scale anomaly score is higher than the anomaly threshold, it is marked as a potential anomaly record; if the multi-scale anomaly score is within the preset gray zone, it can be re-checked later.

7. The method for detecting power anomalies based on big data according to claim 6, characterized in that: Step 301 is followed by step 302. For each user's record, if the multi-scale anomaly score is higher than the anomaly threshold and the long-term and short-term errors are both higher than expected, the record is pre-included in the potential individual anomaly list; otherwise, it is considered a normal level or gray zone record. A group deviation rate is constructed to identify whether a large range of users have a load increase or decrease trend at the same time, so as to distinguish overall pattern shifts from individual anomalies and generate a list of key anomalies and regular anomalies.

8. The method for detecting power anomalies based on big data according to claim 7, characterized in that: The step 4 includes step 401, reviewing the generated key exception list and regular exception list, outputting the annotation information and storing it in the annotation library; Records with slight errors or obvious group characteristics in the regular anomaly list are batch processed and marked as false alarms or real anomalies.

9. The method for detecting power anomalies based on big data according to claim 8, characterized in that: After executing step 401, if the short-term model and the long-term model find new real abnormal or non-abnormal samples, they perform momentum or incremental updates on the drift judgment threshold and the multi-scale abnormality threshold based on the new annotation information, and send the confirmed normal or abnormal samples back to the model training pipeline in the second step.

10. The method for detecting power anomalies based on big data according to claim 9, characterized in that: When the threshold correction value constructed by integration and exponential amplification operations is continuously greater than expected for a period of time, increase or decrease the drift judgment threshold and the multi-scale anomaly threshold. If the threshold correction value remains stable or lower than expected, maintain the current setting. Samples that are marked as false positives but have been judged as abnormal by short-term or long-term models are re-included in the normal training set, real abnormal samples are included in the abnormal training set, and uncertain samples are temporarily stored in the candidate set.

Images