Structured preprocessing system for multi-source heterogeneous energy consumption data for power prediction

Abstract

The present application relates to the technical field of data processing, in particular to a multi-source heterogeneous energy consumption data structured preprocessing system for power prediction, which solves the technical problem that in the prior art, due to the single preprocessing means, the real anomaly in the multi-source data preprocessing process of power prediction is incorrectly processed. The system is used for: acquiring a reference load sequence and an actual load sequence, comparing the actual load sequence with the reference load sequence, and identifying a load deviation interval; determining the explanation ability of each data source to the load deviation interval according to the load deviation between the actual load sequence and the reference load sequence in each load deviation interval; performing a preprocessing action on the global data of each data source within the prediction window according to the corresponding preprocessing action matched according to the explanation ability; and taking the structured data including the global data sequence of each data source after processing and the reference load sequence as a preprocessing result.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a structured preprocessing system for multi-source heterogeneous energy consumption data used for power prediction. Background Technology

[0002] With the deepening of the digital transformation of the power system, load forecasting on the electricity consumption side has become a key supporting technology in energy management, demand response, and grid dispatch, and the requirements for forecast accuracy and robustness are increasing. Traditional power forecasting methods mainly rely on time series modeling based on single historical meter data. However, modern electricity consumption scenarios are highly complex and characterized by multi-factor coupling. Users' electricity consumption behavior is not only affected by the status of their own equipment, but also closely related to multi-source heterogeneous factors such as weather conditions, production plans, electricity price signals, staff scheduling, and holiday arrangements. Single-source forecasting methods based solely on historical power consumption are no longer sufficient to meet the demand for high-precision forecasting.

[0003] To utilize multi-source information, existing power forecasting technologies typically clean and align various data sources, such as electricity meters, equipment sensors, production plans, and weather stations, using a unified format, and then simply concatenate them as input features for the forecasting model. While this method solves the problem of inconsistent data formats, it is prone to data misprocessing in practical applications. For example, it may treat sudden power fluctuations caused by actual equipment startup or shutdown as noise smoothing, mechanically align the impact of weather on power consumption to the same time while ignoring actual response delays, mistakenly identify planned start or end times as boundaries of actual power consumption changes, or treat abnormal jumps caused by data retransmission or communication interruptions as normal power consumption samples. Summary of the Invention

[0004] To address the technical problem of erroneous handling of real anomalies in the multi-source data preprocessing process for power consumption forecasting due to the reliance on single preprocessing methods in existing technologies, this invention aims to provide a structured preprocessing system for multi-source heterogeneous energy consumption data in power consumption forecasting. The specific technical solution adopted is as follows: Firstly, a structured preprocessing system for multi-source heterogeneous energy consumption data used in power generation forecasting is provided, comprising: a deviation analysis module configured to acquire a reference load sequence of the power user within a forecast window, and an actual load sequence within a historical window of equal length under the same operating conditions as the forecast window, and to identify at least one load deviation interval by comparing the actual load sequence with the reference load sequence; an explanatory power quantification module configured to analyze the temporal correlation, trend consistency, and contribution to the load deviation of data segments from each data source within the load deviation interval based on the load deviation between the actual load sequence and the reference load sequence, and to comprehensively determine the explanatory power of each data source for the load deviation interval; a preprocessing execution module configured to comprehensively match corresponding preprocessing actions based on the explanatory power of each data source for at least one load deviation interval, and to perform preprocessing actions on the global data of each data source within the forecast window to obtain a processed global data sequence for each data source; and a sample output module configured to output the structured data, including the processed global data sequence and the reference load sequence of each data source, as the preprocessing result within the forecast window to the power generation forecasting model.

[0005] Based on the above technical solution, in the multi-source heterogeneous energy consumption data structured preprocessing system for power prediction provided by this invention, deviation intervals in the actual load are identified by establishing a reference load, and the temporal correlation, consistency of change trends, and contribution of each data source with load deviation within the deviation interval are analyzed. The interpretability of each data source is comprehensively quantified, and then the corresponding preprocessing actions are matched according to the interpretability to perform targeted processing on the global data. Finally, structured data containing the processed global data sequence and the reference load sequence are output. Thus, causal perception-type structured preprocessing of multi-source heterogeneous energy consumption data is realized in the power prediction scenario. This avoids the defects of traditional unified cleaning methods, such as the missmoothing of real changes, the misalignment of lag relationships, and the misuse of abnormal data, and improves the availability and reliability of the preprocessing results for downstream power prediction models.

[0006] In conjunction with the first aspect above, in one possible implementation, the aforementioned deviation analysis module is specifically configured to: statistically analyze the deviation values ​​between the actual load sequence and the reference load sequence at corresponding time points; acquire load data from multiple historical equal-length windows under the same operating conditions as the prediction window; for each time point, determine the deviation threshold for that time point based on the dispersion of the load data from multiple historical equal-length windows at that time point; determine the time points where the deviation value is greater than the deviation threshold as deviation points, and merge consecutively occurring deviation points into a load deviation interval.

[0007] In conjunction with the first aspect above, in one possible implementation, the aforementioned deviation analysis module is specifically configured to: acquire load data from multiple historical equal-length windows under the same operating conditions as the prediction window, and generate a reference load sequence based on the load statistics of the multiple historical equal-length windows at corresponding time points.

[0008] In conjunction with the first aspect above, in one possible implementation, the aforementioned explanatory power quantification module is specifically configured as follows: The time points when values ​​in a data segment jump or reach extreme values ​​are taken as key change points; the temporal correlation is determined based on the temporal position relationship of each data source's key change point relative to the load deviation interval; the state deviation direction of the statistical data segment relative to the data source's benchmark reference value within the load deviation interval, and the load deviation direction of the actual load sequence relative to the reference load sequence within the load deviation interval, are used to determine the consistency of the change trend based on the state deviation direction and the load deviation direction; a simulated load sequence corresponding to the data segment is generated through a preset response model; the contribution is determined based on the load difference between the actual load sequence and the simulated load sequence within the load deviation interval, and the load deviation between the actual load sequence and the reference load sequence within the load deviation interval; and the explanatory power is obtained by weighted summation of the temporal correlation, the consistency of the change trend, and the contribution.

[0009] In conjunction with the first aspect above, in one possible implementation, the aforementioned explanatory power quantification module is specifically configured as follows: if the timestamp of the key change point is located within the load deviation interval or within a preset time window before the start time of the load deviation interval, then the time correlation is determined to be a first value; otherwise, the time correlation is determined to be a second value; the first value is greater than the second value.

[0010] In conjunction with the first aspect above, in one possible implementation, the aforementioned explanatory power quantification module is specifically configured to: use the proportion of time points in the load deviation interval where the state deviation direction is consistent with the load deviation direction to the total number of time points in the load deviation interval as the consistency of the changing trend.

[0011] In conjunction with the first aspect above, in one possible implementation, the aforementioned explanatory power quantification module is specifically configured as follows: acquiring a preset response model matching the data type of the data source; if the data type is discrete state data, the preset response model is a step response model, and the response amplitude of the step response model is determined based on historical statistical values ​​of load changes before and after state changes in the data segment; if the data type is continuous environmental data, the preset response model is a linear response model, and the response coefficient of the linear response model is determined based on the linear regression coefficient between the change in the data source and historical load changes; inputting the data segment into the preset response model to generate simulated load changes; adding the simulated load changes to the reference load sequence to obtain the simulated load sequence; within the load deviation interval, summing the absolute values ​​of the differences between the actual load sequence and the simulated load sequence as the load difference, and summing the absolute values ​​of the differences between the actual load sequence and the reference load sequence as the load deviation; performing a negative correlation mapping by dividing the load difference by the load deviation to obtain the contribution.

[0012] In conjunction with the first aspect above, in one possible implementation, the aforementioned preprocessing execution module is specifically configured as follows: if a data source corresponds to a load deviation interval, then the corresponding preprocessing action is matched according to the data source's interpretability of the load deviation interval; if a data source corresponds to multiple load deviation intervals, then the corresponding preprocessing action is matched according to the arithmetic mean of the data source's interpretability of the multiple load deviation intervals; the preprocessing actions include transition hold action, hysteresis alignment action, and isolation weight reduction action.

[0013] In conjunction with the first aspect above, in one possible implementation, the aforementioned transition preservation action includes: if the data source data type is discrete state data and the explanatory power or arithmetic mean is greater than a first threshold, then retain the state transition boundary points in the global data and prohibit smoothing or interpolation processing on the transition boundary points; the lag alignment action includes: if the data source data type is continuous environmental data and the explanatory power or arithmetic mean is greater than a second threshold, then use cross-correlation analysis to determine the lag time of the data source relative to the load change, and shift each data point in the global data backward by the lag time; the second threshold is less than the first threshold; the isolation and weight reduction action includes: if the explanatory power or arithmetic mean is less than a third threshold, then remove the data source, or multiply the global data sequence of the data source in the structured data by a reduced weighting coefficient; the third threshold is less than the second threshold.

[0014] In conjunction with the first aspect above, in one possible implementation, the sample output module is further configured to: select the primary data source within the load deviation interval from multiple data sources corresponding to each load deviation interval based on the interpretability of each data source; generate a corresponding simulated load sequence based on the data sequence of the primary data source for each load deviation interval, fill the non-interval areas with a reference load sequence, and obtain a comprehensive load sequence covering the prediction window; determine the sample quality label of the preprocessing result based on the comprehensive load sequence, and output it to the power prediction model.

[0015] The present invention has the following beneficial effects: By establishing a reference load to identify deviation intervals in the actual load, and analyzing the temporal correlation, consistency of change trends, and contribution of each data source with load deviation within the deviation interval, the explanatory power of each data source is comprehensively quantified. Then, based on the explanatory power, corresponding preprocessing actions are matched to perform targeted processing on the global data. Finally, structured data containing the processed global data sequence and the reference load sequence is output. This achieves causal perception-based structured preprocessing of multi-source heterogeneous energy consumption data in the power prediction scenario, avoiding the defects of traditional unified cleaning methods such as incorrect smoothing of real changes, misalignment of lag relationships, and misuse of abnormal data. This improves the usability and reliability of the preprocessing results for downstream power prediction models. Attached Figure Description

[0016] Figure 1 A system structure diagram of a multi-source heterogeneous energy consumption data structured preprocessing system for power prediction is provided in one embodiment of the present invention; Figure 2 The present invention provides a flowchart of a method for structuring and preprocessing multi-source heterogeneous energy consumption data for power prediction, as an embodiment of the present invention. Detailed Implementation

[0017] The following description, in conjunction with the accompanying drawings, details a specific scheme for a multi-source heterogeneous energy consumption data structuring preprocessing system for power prediction provided by the present invention.

[0018] Please see Figure 1 The diagram illustrates a system structure of a multi-source heterogeneous energy consumption data structured preprocessing system for power generation prediction, provided by an embodiment of the present invention. The multi-source heterogeneous energy consumption data structured preprocessing system for power generation prediction includes: a deviation analysis module 1, an explanatory power quantification module 2, a preprocessing execution module 3, and a sample output module 4.

[0019] The deviation analysis module 1, typically deployed in a data server or edge computing gateway, is responsible for acquiring the reference load sequence of the electricity user within the forecast window, as well as the actual load sequence within a historical window of equal length under the same operating conditions as the forecast window. This module identifies at least one load deviation interval by comparing the actual load sequence with the reference load sequence, and records the start and end positions and average deviation magnitude of each interval. The list of load deviation intervals output by the deviation analysis module 1 serves as the analysis target for the subsequent explanatory power quantification module 2.

[0020] The explanatory power quantification module 2 typically runs as a software component within a data analysis platform or real-time stream processing engine. It receives the load deviation intervals output by the deviation analysis module 1, as well as data segments from each candidate external data source within the corresponding intervals. This module quantifies the explanatory power of each data source for each load deviation interval from three dimensions: temporal correlation, consistency of trend, and contribution to load deviation. The quantified values ​​from these three dimensions are then weighted and summed to obtain a comprehensive explanatory power value for each data source. This comprehensive explanatory power value is then transmitted to the preprocessing execution module 3 as the basis for matching preprocessing actions.

[0021] The preprocessing execution module 3 is typically integrated into the data preprocessing pipeline or ETL process. It receives the comprehensive explanatory power value for each data source across each interval from the explanatory power quantification module 2, as well as the global data sequence of each data source within the prediction window. This module synthesizes the explanatory power based on the number (single or multiple) of load deviation intervals corresponding to the data source (e.g., taking the arithmetic mean), and then matches corresponding preprocessing actions according to the data type and the comprehensive explanatory power value. These actions include transition preservation actions to retain discrete state transition boundaries, lag alignment actions to perform time shifting on continuous environmental data, and isolation and weighting actions to remove or reduce the weight of low-explanatory-power data sources. The preprocessing execution module 3 outputs the preprocessed global data sequence for each data source, which is then sent to the sample output module 4.

[0022] The sample output module 4 is implemented as a message queue producer or file writer, receiving the global data sequence processed by all data sources from the preprocessing execution module 3, as well as the reference load sequence generated by the deviation analysis module 1. This module can further select the data source with the largest comprehensive explanatory power value from multiple data sources in each load deviation interval as the master data source, and use the master data source to generate a simulated load sequence, which is then concatenated with the reference load sequence to obtain a comprehensive load sequence covering the entire prediction window. During the training phase, the module determines the sample quality label based on the degree of difference between the comprehensive load sequence and the actual load label sequence; during the prediction phase, this label is set to a null value. Finally, the sample output module 4 sends the structured data, including the global data sequence processed by each data source, the reference load sequence, the load deviation interval location information, the master data source identifier, and the sample quality label, to the downstream power prediction model through a standard interface.

[0023] Please see Figure 2 The diagram illustrates a flowchart of a method for performing structured preprocessing of multi-source heterogeneous energy consumption data for power prediction, according to an embodiment of the present invention. This method includes: S1. Obtain the reference load sequence of the electricity user within the prediction window, and the actual load sequence within the same historical window under the same operating conditions as the prediction window. Compare the actual load sequence with the reference load sequence to identify at least one load deviation interval.

[0024] In the preprocessing stage of electricity forecasting, a benchmark reflecting the normal electricity consumption patterns needs to be established for each electricity user (such as a building or a workshop). Simultaneously, real electricity consumption data from previous periods under the same operating conditions is acquired. By comparing these two data, the time intervals in which significant anomalies or special events in electricity consumption behavior are identified. The reference load sequence represents the expected load of the electricity user under the absence of special disturbances (such as abnormal equipment start-up / shutdown, extreme weather, or production plan adjustments), providing a comparative benchmark. The actual load sequence selects real-world data collected within a historical window of equal length to the forecast window under the same operating conditions (such as a weekday or the same season). This window is located before the forecast window and is used to characterize actual electricity consumption behavior. By comparing the actual load sequence and the reference load sequence point by point on the same time axis, continuous time regions where deviations significantly exceed the normal fluctuation range can be identified. These regions are called load deviation intervals, used to focus the attention of multi-source heterogeneous data analysis and preprocessing on a few key variation intervals, avoiding meaningless calculations on stable data throughout the entire time period.

[0025] In some implementations, load data from multiple historical equal-length windows under the same operating conditions as the prediction window are obtained, and a reference load sequence is generated based on the load statistics of the multiple historical equal-length windows at the corresponding time points.

[0026] Specifically, the start time, time unit (e.g., 15 minutes), and length (e.g., 24 hours, corresponding to 96 time points) of the forecast window are first determined. Based on the operating conditions of the forecast window (e.g., whether it is a weekday, whether it falls within the same season, and whether it avoids holidays), multiple historical dates matching these conditions are selected from the historical electricity database. For each selected historical date, load data within the same relative time period as the forecast window (e.g., 08:00 on one day to 08:00 on the next day) is extracted to form a historical equal-length window sequence. Each historical equal-length window contains the same number of time points as the forecast window, and the time alignment is consistent (i.e., the first time point of each historical equal-length window corresponds to the first time point of the forecast window). For each time point, the load statistics for multiple historical equal-length windows at that location are calculated, specifically using the arithmetic mean or median. Arranging the load statistics for all time points in chronological order yields a complete reference load sequence. This sequence represents the typical load pattern of the electricity-consuming entity under the same operating conditions, and its smoothness can offset the impact of occasional events on a single day.

[0027] In some implementations, the deviation between the actual load sequence and the reference load sequence at corresponding time points is statistically analyzed. Specifically, for each time point, the absolute value of the difference between the actual load value and the reference load value is calculated.

[0028] The deviation threshold is determined based on the dispersion of load data from multiple historical equal-length windows at corresponding time points. Specifically, for each time point, load data from multiple historical equal-length windows at that time point are extracted, and the standard deviation of these horizontal historical data is calculated. This standard deviation reflects the normal fluctuation range of that specific time point under the condition of no special events. Then, this standard deviation is multiplied by a preset coefficient as the deviation threshold for that time point. This coefficient is set empirically. For example, for general office buildings, the coefficient can be 1.5; for industrial production scenarios with large load fluctuations, the coefficient can be 2.0. The larger the coefficient, the stricter the deviation judgment and the fewer deviation intervals are identified; the smaller the coefficient, the easier it is to identify deviation intervals. In this way, each time point has an independent deviation threshold, which can adaptively reflect the normal fluctuation range of that time point and avoid interference from the daily peak-valley range on the deviation judgment. Further, the obtained deviation threshold is compared with a preset noise floor limit, and the larger of the two is taken as the preprocessing deviation threshold for that time point. The noise floor limit is set according to the minimum acquisition accuracy of the data source or the physical anti-shake tolerance.

[0029] The time point where the deviation value is greater than the deviation threshold is determined as the deviation point. After marking all time points, consecutive deviation points are merged into a load deviation interval.

[0030] S2. Based on the load deviation between the actual load sequence and the reference load sequence within each load deviation interval, analyze the temporal correlation, consistency of change trend, and contribution of each data source to the load deviation within the load deviation interval, and comprehensively determine the explanatory power of each data source for the load deviation interval.

[0031] After identifying at least one load deviation interval, it is necessary to quantify the explanatory power of each data source (such as equipment status, outdoor temperature, electricity price signals, etc.) for abnormal load changes within that interval. This is because the causal relationship between different data sources and load changes varies in strength. Some data source changes closely match the load deviation in time, some change in the same direction, and some can simulate most of the load deviation through physical models. Relying on any single dimension alone may lead to misjudgment or omission. Therefore, this step evaluates the data source from three dimensions: temporal correlation, which determines whether the key changes in the data source occur within a reasonable time window of the load deviation; consistency of change trend, which determines whether the direction of change in the data source is logically consistent with the direction of the load deviation; and contribution, which determines what proportion of the load deviation the data source can simulate through a preset response model. Finally, the quantified values ​​of the three dimensions are combined to obtain a comprehensive explanatory power.

[0032] In some implementations, for discrete data sources (such as the power-on / off status of equipment, or peak-valley fluctuations in electricity prices), the critical change point refers to the moment when a data value undergoes a sudden change, i.e., the instant it jumps from one stable value to another. Based on this, the data segment is traversed. If a value in the data segment jumps, i.e., the difference between the values ​​of two adjacent time points is greater than a preset state change threshold (usually half the absolute value of the difference between the data values ​​of two extreme states), then the latter time point is the critical change point. Alternatively, for continuous data sources (such as outdoor temperature and humidity), the time point that reaches an extreme value, i.e., the value at that time point is greater than the values ​​of the adjacent time points (local peak) or less than the values ​​of the adjacent time points (local valley), then the extreme value point is considered the critical change point. Specifically, if the data segment of a continuous data source exhibits monotonic changes and has no extreme value points, then the starting point, ending point, or point with the maximum rate of change of that data segment is selected as the critical change point.

[0033] After obtaining the timestamps of key change points, the temporal correlation is determined based on the temporal position of the key change points of each data source relative to the load deviation interval. Specifically, if the timestamp of the key change point is within the load deviation interval or within a preset time window before the start time of the load deviation interval, the temporal correlation is determined to be a first value; otherwise, it is determined to be a second value. The first value is greater than the second value to reflect that data sources with reasonable temporal matching have higher interpretability. In one specific implementation, the first value is 1 and the second value is 0. This binarization process ensures that only data sources that change within a reasonable time window can obtain a positive contribution in the time dimension. The preset time window is used to accommodate the lead time by which the data source affects the load. Its length is set according to the data source type: for equipment status data, the lead time is usually zero or one time unit because equipment start-up and shutdown affect the load almost instantaneously; for meteorological temperature data, considering the building's thermal inertia, the preset time window can be set to 2 to 4 hours; for production plan data, the preset time window can be set to 1 hour before the plan start time.

[0034] The system identifies the direction of deviation of statistical data segments from the data source's baseline reference value within the load deviation interval, and the direction of load deviation of the actual load sequence from the reference load sequence within the load deviation interval. The consistency of the changing trend is determined based on the direction of deviation and the direction of load deviation. Specifically, firstly, a data source state deviation direction sequence and a load deviation direction sequence are constructed: For each time point within the load deviation interval, the data source's baseline reference value is obtained (using the same set of historical equal-length windows as the reference load sequence, the statistical values ​​of the data source at that time point, such as the median or average, representing the typical value of the data source under the same operating conditions). The difference between the actual data source value and its baseline reference value is calculated. If the difference is positive, the state deviation direction at that time point is marked as "positive deviation"; if it is negative, it is marked as "negative deviation"; if it is zero, it is marked as "no deviation". Simultaneously, the difference between the actual load value and the reference load value at that time point is calculated. If the difference is positive, the load deviation direction is marked as "positive deviation" (i.e., actual load is higher than the baseline); if it is negative, it is marked as "negative deviation" (actual load is lower than the baseline); if the difference is zero, it is marked as "no deviation". Then, the two directional markers are compared point by point. For typical positively correlated data sources (such as air conditioning power increasing with temperature), the positive deviation of the data source should be consistent with the positive deviation of the load, and the negative deviation should be consistent with the negative deviation of the load. For negatively correlated data sources (such as photovoltaic power generation decreasing with cloud cover), the positive deviation of the data source should be consistent with the negative deviation of the load. Each data source can be pre-configured with its correlation symbol. The proportion of time points in the load deviation interval where the state deviation direction is consistent with the load deviation direction to the total number of time points in the load deviation interval is used as the consistency of the trend. The value is in the range [0, 1]. The closer the value is to 1, the higher the consistency between the trend of the data source and the direction of the load deviation; the closer the value is to 0, the more inverse the relationship. This calculation method allows persistent environmental factors (such as continuous high temperatures) to correctly match the positive deviation of the load even if the rate of change is small, due to their continuous positive deviation, thus obtaining a high consistency score.

[0035] The simulated load sequence corresponding to the data segment is generated by a preset response model. The contribution is determined based on the load difference between the actual load sequence and the simulated load sequence within the load deviation interval, and the load deviation between the actual load sequence and the reference load sequence within the load deviation interval.

[0036] Specifically, a preset response model matching the data source data type is obtained. If the data type is discrete state data (e.g., equipment start-up / stop, switch action, electricity price level change), the preset response model is a step response model. This model assumes that when the data source state changes, the load will immediately produce a fixed change, which continues until the state reverses again. The response amplitude of the step response model is determined based on the historical statistical values ​​of the load change before and after the state change in the data segment. Specifically, events where the data source experiences a state change independently without other interference can be retrieved from the historical database. The load change before and after each event is extracted, and the arithmetic mean or median of these changes is calculated as the response amplitude of the data source.

[0037] If the data type is continuous environmental data (e.g., outdoor temperature, humidity, light intensity), the default response model is a linear response model. This model assumes a linear relationship between load deviation and the deviation of the data source from its reference value, and the proportionality coefficient is the response coefficient. The response coefficient of the linear response model is determined based on the linear regression coefficient between the change in the data source and the historical load change. Specifically, historical data under the same operating conditions over a period of time (e.g., the past 60 days) can be selected. The actual value of the data source minus its reference value during that period (which can be the average value of the data source during the baseline period) is used as the independent variable. The variance of the corresponding sequence of the independent variable is evaluated. If the variance of the independent variable is zero, the response coefficient is directly set to zero; if the variance of the independent variable is not zero, a univariate linear regression is performed to obtain the response coefficient. The actual load minus the reference load is used as the dependent variable, and a univariate linear regression is performed. The resulting regression coefficient is the response coefficient.

[0038] After determining the response model and its parameters, the data fragments are input into the preset response model, which outputs the simulated load change at each time point. The simulated load change is then added to the reference load sequence to obtain the simulated load sequence. The simulated load sequence represents the load pattern that should exist if the data source were the sole cause of the load deviation.

[0039] Within the load deviation range, the sum of the absolute values ​​of the differences between the actual load sequence and the simulated load sequence is used as the load difference quantity, and the sum of the absolute values ​​of the differences between the actual load sequence and the reference load sequence is used as the load deviation quantity. The load deviation quantity reflects the total magnitude of the original deviation, while the load difference quantity reflects the remaining unexplained magnitude after interpretation by the data source. The ratio of the load difference quantity to the load deviation quantity is calculated, and this ratio takes a value in the range [0, 1]. The smaller the value, the more fully the data source interprets the load. The contribution degree is obtained by negatively correlated mapping of the comparison values, so that the smaller the ratio, the closer the contribution degree is to 1, and the larger the ratio, the closer the contribution degree is to zero. A feasible negative correlation mapping method is to subtract the ratio from 1 to obtain the contribution degree. Specifically, if the contribution degree calculation result is negative (the simulated load difference quantity is greater than the original load deviation quantity), it indicates that the data source provides a reverse or incorrect interpretation, and the contribution degree can be set to zero. If the load deviation quantity is zero, it means that there is actually no interpretable deviation between the actual load and the reference load within the load deviation range, and the contribution degree can be set to zero.

[0040] The explanatory power of each data source for each load deviation interval is obtained by weighted summation of temporal correlation, trend consistency, and contribution. The weighting principle is as follows: contribution is the ultimate reflection of explanatory power, because even if temporal correlation and trend consistency are satisfied, if the residual after simulation response is still large, the actual explanatory value of the data source is limited. Temporal correlation and trend consistency are auxiliary conditions used to eliminate spurious correlations due to time mismatch or incorrect direction. Therefore, the contribution weight is usually set to a large value, such as setting the weight of temporal correlation to 0.3, the weight of trend consistency to 0.3, and the weight of contribution to 0.4. The calculated explanatory power is in the interval [0, 1].

[0041] S3. Based on the interpretability of each data source for at least one load deviation interval, match the corresponding preprocessing action, perform the preprocessing action on the global data of each data source within the prediction window, and obtain the global data sequence after processing for each data source.

[0042] It is important to note that this step involves performing actions on the global data because preprocessing actions (such as shifting the entire sequence or preserving all transition boundaries) should be consistently applied across the entire time range; otherwise, inconsistent representations of the same data source will occur in different intervals.

[0043] After obtaining the comprehensive explanatory power of each data source for each load deviation interval, it is necessary to determine the most suitable preprocessing strategy for each data source. Different data sources have different types of causal relationships with load changes, specifically including: discrete state data (such as equipment start-up and shutdown) usually have an instantaneous and step-like impact on the load. Smoothing would erase key change boundaries, so "turn-off preservation" is required; continuous environmental data (such as temperature) often has a delayed impact on the load, so "lag alignment" is needed to shift the data sequence on the time axis to match the true causal time sequence; and for data sources with very low or even negative explanatory power, they not only do not help with prediction but may also introduce noise, so they should be eliminated or have their weight reduced through "isolation and weight reduction".

[0044] In some implementations, if a data source corresponds to one load deviation interval, the corresponding preprocessing action is matched based on the data source's explanatory power for the load deviation interval; if a data source corresponds to multiple load deviation intervals (for example, outdoor temperature affects both the morning load increase interval and the afternoon load decrease interval), the corresponding preprocessing action is matched based on the arithmetic mean of the data source's explanatory power for multiple load deviation intervals (representing the overall explanatory power of the data source for all load deviations within the entire forecast window).

[0045] Preprocessing actions include transition hold actions, hysteresis alignment actions, and isolation deweighting actions.

[0046] Specifically, the triggering condition for the transition hold action is as follows: the data source data type is discrete state data (e.g., equipment on / off status, electricity price level, production plan execution flag), and the explanatory power or arithmetic mean is greater than a first threshold. The first threshold is a preset value used to filter discrete data sources with sufficiently high explanatory power. Only discrete events that can strongly explain load deviations are worth retaining their transition boundaries, for example, set to 0.6. If the explanatory power is weak, retaining the boundary may introduce false features. When the condition is met, the transition hold action is executed: the state transition boundary point (i.e., the position where the difference between adjacent time points exceeds the state change threshold) is retained in the global data, and smoothing or interpolation processing of the transition boundary point is prohibited. For example, if the equipment status changes from off to on, the state value at that time point should remain on, and should not be corrected by the filtering algorithm due to excessive difference from the previous and subsequent values. At the same time, transition markers can be added to the generated sample features. Through this processing, the true time sequence relationship of discrete events having a step impact on the load is retained, avoiding the mistaken smoothing of real electricity consumption changes as noise.

[0047] The triggering conditions for the lag alignment action are: the data source is continuous environmental data (e.g., outdoor temperature, humidity, light intensity), and its explanatory power or arithmetic mean is greater than a second threshold, which is less than the first threshold. This is because even with moderate explanatory power, continuous environmental data can still improve its match with the load through time shifting, while discrete events require higher explanatory power to justify retaining jumps, for example, a value of 0.5. When the conditions are met, the lag alignment action is executed: cross-correlation analysis is used to determine the lag time of the data source relative to load changes. Specifically, the initial data sequence and the load deviation sequence (the difference between the actual load and the baseline) of the data source within a historical period (e.g., several past days) are taken, and the correlation coefficient at different time offsets is calculated. For example, the initial data sequence is shifted forward or backward relative to the load deviation sequence by several time units (e.g., 4 hours, with time units as the step size), and the Pearson correlation coefficient at each offset is calculated. The offset that maximizes the correlation coefficient is selected as the lag time. If the offset is positive, it indicates that the data source change precedes the load change (i.e., the data source changes first, and the load responds later), in which case the data source needs to be shifted backward; if it is negative, it needs to be shifted forward. In practice, the most common scenario is a delayed response (e.g., the load only increases after a period of time following a temperature rise), in which case the lag time is positive. Then, each data point in the global dataset is shifted backward by the lag time. After shifting, the first few time points of the sequence will be missing (e.g., after shifting 2 hours, the first two time points will have no data). The missing parts are filled forward (using the first valid value after shifting) or by the historical average; the parts exceeding the end of the window are discarded. This shifting operation ensures that the changes in the data source and the load response are correctly aligned in time, solving the problem of traditional methods mechanically aligning to the same moment while ignoring physical delays.

[0048] The isolation and weighting action is triggered when the explanatory power or arithmetic mean is less than a third threshold, and the third threshold is less than a second threshold. This is used to exclude data sources with extremely low or even negative explanatory power, which not only do not help with prediction but may also interfere with model learning; for example, a threshold of 0.3 is set. When the condition is met, the isolation and weighting action is executed: the data source is removed, or in structured data, the global data sequence of the data source is multiplied by a reduced weighting coefficient (e.g., 0.1) to reduce the contribution of that dimension. This action avoids the negative impact of low-quality data sources on the prediction model while retaining the possibility of fine-tuning near the threshold boundary.

[0049] The first, second, and third thresholds described above satisfy a strict decreasing relationship: the first threshold is greater than the second threshold, and the second threshold is greater than the third threshold. Specific values ​​can be adjusted based on the application scenario, prediction accuracy requirements, and data source characteristics. To more actively preserve discrete event boundaries, the first threshold can be lowered; to encourage more continuous data sources to participate in lag alignment, the second threshold can be lowered; and to reduce the number of excluded data sources, the third threshold can be lowered.

[0050] S4. The structured data, including the global data sequence and reference load sequence processed by each data source, is output as the preprocessing result within the prediction window to the power prediction model.

[0051] After the aforementioned steps, each data source obtains a global data sequence that has undergone targeted preprocessing within the prediction window, as well as a reference load sequence covering the entire prediction window. These data are packaged into a structured data object, which contains both the input features required for the model to perform load prediction (i.e., the sequences processed by each data source) and benchmark information for the model to refer to (i.e., the reference load sequence). This object is then output to the downstream power prediction model, enabling the model to learn the mapping relationship between features and load during training and to output load prediction values ​​based on the input features during prediction.

[0052] In some implementations, structured data can be organized in a key-value pair format (e.g., JSON objects), where the keys are data source identifiers (e.g., "outdoor temperature," "air conditioner operating status"), and the values ​​are the corresponding preprocessed sequences. Additionally, a dedicated key (e.g., "reference_load") is used to store the reference load sequence. This structured data is sent to the power prediction model via a standard data interface (e.g., a message queue). Upon receiving the data, the model can parse the feature sequences from each data source and combine them with the reference load sequence for load prediction or training.

[0053] In some implementations, this method further includes selecting the primary data source within each load deviation interval from multiple data sources based on the explanatory power of each data source. Specifically, from the multiple data sources corresponding to each load deviation interval, the data source with the highest explanatory power value is selected as the primary data source for that interval, representing the most significant cause of the load change in that interval. For example, if the load deviation interval in the morning is affected by both air conditioning operation and outdoor temperature rise, but the explanatory power of air conditioning operation (0.85) is higher than that of temperature rise (0.60), then the air conditioning status is selected as the primary data source for that interval. If two data sources have equal and maximum explanatory power, either one can be arbitrarily selected as the primary data source. The primary data source selection is performed independently for each load deviation interval; therefore, the primary data sources for different intervals can be the same or different.

[0054] A corresponding simulated load sequence is generated based on the data sequence from the primary data source for each load deviation interval. Specifically, for the primary data source, using the preset response model determined in S2 that matches the data source type, the data segment of the corresponding interval within the prediction window is input into the model to generate a simulated load change. This simulated load change is then added to the value of the reference load sequence at the same time point to obtain the simulated load sequence within that interval, reflecting the load change pattern that should occur if only the primary data source were active. In non-interval areas, since there are no abnormal changes requiring special explanation, the reference load sequence is directly used to fill the gaps. The simulated load sequences of all intervals are concatenated with the reference load values ​​of the non-interval areas in chronological order to obtain a comprehensive load sequence covering the prediction window.

[0055] The preprocessed sample quality label is determined based on the comprehensive load sequence and output to the power prediction model. The sample quality label quantifies the degree of load fluctuation in the current sample that cannot be explained by any data source; its calculation is performed only during the training phase (because the actual load label sequence corresponding to the prediction window can be obtained during training). Specifically, the system obtains the actual load label sequence within the prediction window (i.e., historically collected actual load values, used as the ground truth for supervised learning) and the comprehensive load sequence generated in the previous step. The degree of difference between the two is calculated, typically using the root mean square error (RMSE) as a metric. The smaller the value, the more the currently available data source can explain most of the load changes, and the higher the sample quality; the larger the value, the more load fluctuations that are not explained by any data source (possibly from uncollected data, measurement noise, or complex coupling effects), and the lower the sample quality. If, during the prediction phase (i.e., when the actual load label sequence does not exist), the sample quality label is set to null, and the downstream model can skip the weighted calculation based on this identifier. The system adds the calculated sample quality label as an additional field to the structured data and outputs it to the power prediction model. During training, the model can use these labels to weight samples: high-quality samples are assigned higher weights, and low-quality samples are assigned lower weights, preventing the model from learning unreliable patterns. For example, when training an electricity prediction model, the preprocessing system generates sample quality labels of 0.95 and 0.12 for two samples, respectively. 0.95 corresponds to samples fully explained by the data source, while 0.12 corresponds to samples with a large amount of unexplained fluctuations. When calculating the loss, the model multiplies the original prediction error of the first sample by 0.95 and the error of the second sample by 0.12, then averages them to obtain the final loss. This ensures that high-quality samples contribute significantly more to model parameter updates than low-quality samples, guiding the model to prioritize learning electricity consumption patterns that can be reasonably explained by the data source and avoiding fitting noise.

[0056] The final structured output data can include the following fields: the global data sequence processed by each data source, the reference load sequence, the location information of each load deviation interval (start time index and end time index), the main data source identifier and its explanatory power value for each load deviation interval, and the sample quality label. This output not only meets the basic requirements of the prediction model for input features but also provides rich meta-information for more complex modeling (e.g., using the main data source identifier to implement an attention mechanism, and using the sample quality label to implement sample weighting or confidence output). Through the above steps, the structured preprocessing of multi-source heterogeneous energy consumption data is completed, transforming the original messy data into a complete, quality-evaluable, and causally interpretable sample, laying a solid foundation for high-precision prediction by the power consumption prediction model.

[0057] Based on the above technical solution, deviation intervals in the actual load are identified by establishing a reference load. The temporal correlation, consistency of change trends, and contribution of each data source with the load deviation within the deviation interval are analyzed. The explanatory power of each data source is comprehensively quantified. Then, according to the explanatory power, corresponding preprocessing actions are matched to perform targeted processing on the global data. Finally, structured data containing the processed global data sequence and the reference load sequence are output. Thus, causal perception-type structured preprocessing of multi-source heterogeneous energy consumption data is realized in the power prediction scenario. This avoids the defects of traditional unified cleaning methods, such as the missmoothing of real changes, the misalignment of lag relationships, and the misuse of abnormal data. It improves the availability and reliability of the preprocessing results for downstream power prediction models.

Claims

1. A structured preprocessing system for multi-source heterogeneous energy consumption data used for power prediction, characterized in that, include: The deviation analysis module is configured to acquire a reference load sequence of the electricity user within a prediction window, and an actual load sequence within a historical window of the same length as the prediction window under the same operating conditions, and to identify at least one load deviation interval by comparing the actual load sequence with the reference load sequence. The explanatory power quantification module is configured to analyze the temporal correlation, trend consistency, and contribution of each data source to the load deviation between the actual load sequence and the reference load sequence within each load deviation interval, and comprehensively determine the explanatory power of each data source for the load deviation interval. The preprocessing execution module is configured to comprehensively match the corresponding preprocessing action based on the interpretability of each data source for the at least one load deviation interval, and perform preprocessing action on the global data of each data source within the prediction window to obtain the global data sequence after processing for each data source. The sample output module is configured to output structured data, including the global data sequence processed by each data source and the reference load sequence, as a preprocessing result within the prediction window to the power prediction model; The explanatory power quantification module is specifically configured as follows: The time points when the values ​​in the data segment jump or reach extreme values ​​are taken as key change points. The time correlation is determined based on the time position relationship between the key change points of each data source and the load deviation interval. The system calculates the state deviation direction of the data segment relative to the data source baseline reference value within the load deviation interval, and the load deviation direction of the actual load sequence relative to the reference load sequence within the load deviation interval. Based on the state deviation direction and the load deviation direction, it determines the consistency of the change trend. The simulated load sequence corresponding to the data segment is generated by a preset response model. The contribution is determined based on the load difference between the actual load sequence and the simulated load sequence within the load deviation interval, and the load deviation between the actual load sequence and the reference load sequence within the load deviation interval. The explanatory power is obtained by weighting and summing the temporal correlation, the consistency of the trend of change, and the contribution.

2. The multi-source heterogeneous energy consumption data structured preprocessing system for power prediction according to claim 1, characterized in that, The deviation analysis module is specifically configured as follows: Calculate the deviation between the actual load sequence and the reference load sequence at corresponding time points; Obtain load data from multiple historical equal-length windows under the same operating conditions as the prediction window. For each time point, determine the deviation threshold for that time point based on the dispersion of the load data from the multiple historical equal-length windows at that time point. The time point when the deviation value is greater than the deviation threshold is determined as the deviation point, and consecutive deviation points are merged into a load deviation interval.

3. The multi-source heterogeneous energy consumption data structured preprocessing system for power prediction according to claim 1, characterized in that, The deviation analysis module is specifically configured as follows: Obtain load data from multiple historical equal-length windows under the same operating conditions as the prediction window, and generate the reference load sequence based on the load statistics of the multiple historical equal-length windows at corresponding time points.

4. The multi-source heterogeneous energy consumption data structured preprocessing system for power prediction according to claim 1, characterized in that, The explanatory power quantification module is specifically configured as follows: If the timestamp of the key change point is located within the load deviation interval or within a preset time window before the start time of the load deviation interval, the time correlation is determined to be a first value; otherwise, the time correlation is determined to be a second value; the first value is greater than the second value.

5. The multi-source heterogeneous energy consumption data structured preprocessing system for power prediction according to claim 1, characterized in that, The explanatory power quantification module is specifically configured as follows: The consistency of the change trend is determined by the proportion of time points within the load deviation interval where the state deviation direction is consistent with the load deviation direction to the total number of time points within the load deviation interval.

6. The multi-source heterogeneous energy consumption data structured preprocessing system for power prediction according to claim 1, characterized in that, The explanatory power quantification module is specifically configured as follows: Obtain a preset response model that matches the data type of the data source; if the data type is discrete state data, the preset response model is a step response model, and the response amplitude of the step response model is determined based on the historical statistical values ​​of the load change before and after the state change in the data segment; if the data type is continuous environmental data, the preset response model is a linear response model, and the response coefficient of the linear response model is determined based on the linear regression coefficient between the change in the data source and the historical load change. The data fragment is input into the preset response model to generate a simulated load change, and the simulated load change is added to the reference load sequence to obtain a simulated load sequence. Within the load deviation range, the sum of the absolute values ​​of the differences between the actual load sequence and the simulated load sequence is used as the load difference, and the sum of the absolute values ​​of the differences between the actual load sequence and the reference load sequence is used as the load deviation. The contribution degree is obtained by negatively mapping the ratio of the load difference to the load deviation.

7. The multi-source heterogeneous energy consumption data structured preprocessing system for power prediction according to claim 1, characterized in that, The preprocessing execution module is specifically configured as follows: If a data source corresponds to one load deviation interval, the corresponding preprocessing action is matched according to the data source's ability to interpret the load deviation interval; if a data source corresponds to multiple load deviation intervals, the corresponding preprocessing action is matched according to the arithmetic mean of the data source's ability to interpret the multiple load deviation intervals. The preprocessing actions include transition hold actions, hysteresis alignment actions, and isolation deweighting actions.

8. The multi-source heterogeneous energy consumption data structured preprocessing system for power prediction according to claim 7, characterized in that, The transition retention action includes: if the data source data is discrete state data and the explanatory power or the arithmetic mean is greater than a first threshold, then the state transition boundary point is retained in the global data and smoothing or interpolation processing of the transition boundary point is prohibited. The lag alignment action includes: if the data source is continuous environmental data and the explanatory power or the arithmetic mean is greater than a second threshold, then cross-correlation analysis is used to determine the lag time of the data source relative to the load change, and each data point in the global data is shifted backward by the lag time; the second threshold is less than the first threshold; The isolation and weight reduction action includes: if the explanatory power or the arithmetic mean is less than a third threshold, then removing the data source, or multiplying the global data sequence of the data source in the structured data by a weight reduction coefficient; the third threshold is less than the second threshold.

9. The multi-source heterogeneous energy consumption data structured preprocessing system for power prediction according to claim 1, characterized in that, The sample output module is also configured to: From multiple data sources corresponding to each load deviation interval, select the primary data source within the load deviation interval based on the interpretability of each data source; A corresponding simulated load sequence is generated based on the data sequence of the main data source for each load deviation interval, and the reference load sequence is used to fill the non-interval areas to obtain a comprehensive load sequence covering the prediction window; The sample quality label of the preprocessed result is determined based on the comprehensive load sequence and output to the power prediction model.

Images