Multi-source heterogeneous data adaptive processing method and system for photovoltaic power prediction

Abstract

This invention discloses an adaptive processing method and system for multi-source heterogeneous data for photovoltaic power prediction. The method includes: adaptively selecting a time reference source and performing spatiotemporal alignment; dynamically detecting abnormal data based on a context-aware mechanism; selecting a repair algorithm from a strategy library based on features such as anomaly type and missing span; evaluating data quality based on an indicator system bound to prediction error sensitivity; and optimizing the preceding processing steps through a parameter callback mechanism when the quality is substandard. The method also records the entire lifecycle of data for traceability. By constructing an adaptive closed-loop data processing mechanism, this invention significantly improves the quality, reliability, and transparency of the prediction input data, providing data support for high-precision photovoltaic prediction.

Description

Technical Field

[0001] This invention relates to the field of new energy power generation prediction technology, and in particular to a method and system for adaptive processing of multi-source heterogeneous data for photovoltaic power prediction. Background Technology

[0002] Driven by the "dual-carbon" strategic goal, the penetration rate of new energy sources, represented by photovoltaic power generation, in the power system continues to rise. To ensure the safe and stable operation of the power grid and improve the absorption of new energy, high-precision photovoltaic power forecasting has become a core technical support for key aspects such as grid dispatching and electricity market transactions. The accuracy of photovoltaic power forecasting is highly dependent on the quality of the input data; therefore, high-quality preprocessing of the multi-source heterogeneous data used in the forecasting model is a prerequisite for improving forecasting performance. Currently, mainstream technical approaches typically integrate multi-source data such as meteorological observation data, numerical weather prediction (NWP), satellite remote sensing, and station monitoring and data acquisition (SCADA) systems, and process the raw data through data cleaning, interpolation, and spatiotemporal alignment to construct an effective input feature set for the forecasting model.

[0003] However, existing data processing methods still have many shortcomings when dealing with the complex data characteristics of photovoltaic systems. First, at the data fusion level, facing data streams from diverse sources and with varying spatiotemporal scales, simple resampling and fixed benchmark alignment methods struggle to balance timeliness and information fidelity, easily leading to the loss of key temporal features or the introduction of artifacts. Second, in anomaly detection, methods based on fixed thresholds or single statistical rules cannot adapt to the inherent non-stationarity and high volatility of photovoltaic systems, often resulting in misjudgments or omissions of normal fluctuations and real data anomalies due to the lack of contextual information (such as sudden weather changes or equipment maintenance). Third, for detected abnormal data, single repair strategies such as linear interpolation are often used. This approach ignores the differences in anomaly types, missing spans, and data fluctuation characteristics, resulting in poor repair effects and potentially introducing new errors. Furthermore, existing technologies generally lack a quantitative evaluation system for the quality of processed data and a data traceability mechanism throughout the entire process, leading to a "black box" approach to data processing. When prediction accuracy declines, it is difficult to quickly locate the root cause of the problem and form a closed-loop optimization, thus limiting the continuous improvement of prediction system performance. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides an adaptive processing method and system for multi-source heterogeneous data for photovoltaic power prediction, to solve the problems mentioned in the background art.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: an adaptive processing method for multi-source heterogeneous data for photovoltaic power prediction, comprising: S1: Access raw data from multiple data sources, and perform format conversion, unit unification, and encoding standardization on the raw data to generate a standardized data stream; S2: Calculate the sampling frequency stability, data availability, and cross-source latency consistency index of each data source, and select the data source with the highest comprehensive score as the time reference source based on the weighted scoring mechanism. Perform time alignment on the standardized data stream according to the target time resolution, and perform spatial interpolation reconstruction based on spatial correlation to generate a spatiotemporally aligned dataset. S3: Based on the sliding window dynamic statistical features and multi-dimensional context awareness mechanism, perform anomaly detection on the spatiotemporal aligned dataset, identify and label multiple anomaly types, and generate an anomaly labeling dataset; S4: Calculate the comprehensive fit score of each candidate strategy in the repair strategy library based on the anomaly type, missing span, fluctuation degree and spatial correlation or similarity of similar days, and select the strategy with the highest comprehensive fit score to perform repair on the abnormal data in the anomaly labeled dataset. S5: Based on the quality index system bound to the sensitivity of photovoltaic power prediction error, calculate the comprehensive data quality score of the repaired data; when the comprehensive data quality score is lower than the preset threshold, trigger the parameter callback mechanism to adjust the weight of the time base, the candidate data source set, or the anomaly detection threshold, or the priority of the repair strategy. S6: Records the entire lifecycle of data processing from input to output, generating data processing records that include processing operations, parameter configurations, and quality change trajectories for source tracing analysis of data quality issues.

[0007] As a preferred embodiment of the multi-source heterogeneous data adaptive processing method for photovoltaic power prediction described in this invention, wherein: the data source access and standardization in S1 includes: By adapting to and parsing raw data packets from various data sources using multiple data access protocols, valid data fields are extracted. Establish a data source metadata registry to record characteristic parameters such as sampling frequency and data precision of each data source; Convert timestamps of different formats to the Coordinated Universal Time (UTC) standard time format; Convert physical quantities of different units into the International System of Units (SI); Assign a globally unique identifier to each data record and cache standardized data in separate streams according to data source type and time window.

[0008] As a preferred embodiment of the multi-source heterogeneous data adaptive processing method for photovoltaic power prediction described in this invention, wherein: the time alignment and spatial interpolation reconstruction in S2 include: For data with a sampling frequency higher than the target time resolution, downsampling aggregation is used; For data with a sampling frequency lower than the target time resolution, a hierarchical interpolation strategy is adopted based on the time span of missing data, and linear interpolation, cubic spline interpolation, or similar day matching interpolation are selected for processing. Based on the spatial topological relationship of each measuring point, the inverse distance weighting method is used to interpolate the spatially sparse points; For raster data, bilinear interpolation is used to unify spatial resolution.

[0009] As a preferred embodiment of the multi-source heterogeneous data adaptive processing method for photovoltaic power prediction described in this invention, wherein: the anomaly detection in S3 includes: Calculate the statistical characteristics of the data within a multi-scale sliding window and establish a dynamic baseline for the statistical characteristics; Construct a multi-dimensional contextual feature vector that includes time context, meteorological context, and equipment context; The threshold used for anomaly detection is dynamically adjusted based on the context feature vector. A multi-rule joint judgment mechanism is adopted. When data triggers multiple abnormal rules at the same time, the abnormality type is comprehensively determined, and the detected abnormalities are marked as missing, out-of-limit, mutation, drift, or frozen.

[0010] As a preferred embodiment of the adaptive processing method for multi-source heterogeneous data for photovoltaic power prediction described in this invention, wherein: the context-aware anomaly detection in S3 includes: Obtain the time context features of the current data point, including time, date type, and season; Obtain the meteorological context features of the current data point, including weather type and cloud cover level; Obtain the device context features of the current data point, including device operating status and historical fault records; Based on the aforementioned time context features, meteorological context features, and equipment context features, the anomaly detection threshold is adjusted together, and a multi-rule joint decision is executed to output the anomaly detection result.

[0011] As a preferred embodiment of the adaptive processing method for multi-source heterogeneous data for photovoltaic power prediction described in this invention, the repair strategy library in S4 includes the following strategies: Linear interpolation repair, spline interpolation repair, similar day replacement repair, physical constraint repair, nearest neighbor point weighted repair, and exponential smoothing repair; Each repair strategy is configured with its applicable conditions, parameter range, and priority.

[0012] As a preferred embodiment of the multi-source heterogeneous data adaptive processing method for photovoltaic power prediction described in this invention, wherein: the quality score evaluation and parameter callback in S5 includes: The timing alignment error index, irradiance-power consistency residual index, and prediction window availability index were calculated, and the comprehensive data quality score was calculated using the analytic hierarchy process. The weight of each index was determined based on its impact on the accuracy of photovoltaic power prediction. When the overall data quality score is lower than the preset threshold and the time alignment error index contributes the most to the low score, increase the weight of the sampling frequency stability index in the time reference selection. When the overall data quality score is lower than the preset threshold and the irradiance-power consistency residual index contributes the most to the low score, the set of candidate data sources for the expanded time base is selected. If the data quality still does not meet the requirements after repair, adjust the priority order of each strategy in the repair strategy library.

[0013] Furthermore, this invention also provides a multi-source heterogeneous data adaptive processing system for photovoltaic power prediction, the system comprising: The multi-source data access and standardization module is configured to: access raw data from multiple data sources, and perform format conversion, unit unification, and encoding standardization on the raw data to generate a standardized data stream; The spatiotemporal alignment and interpolation reconstruction module is configured to perform the following: calculate the sampling frequency stability, data availability, and cross-source latency consistency indicators of each data source; select the data source with the highest comprehensive score as the time reference source based on a weighted scoring mechanism; perform time alignment of the standardized data stream according to the target time resolution; perform spatial interpolation reconstruction based on spatial correlation; and generate a spatiotemporal aligned dataset, which includes: The adaptive time reference selection unit is configured to calculate and select a time reference source based on three types of indicators: sampling frequency stability, data availability, and cross-source latency consistency. The hierarchical temporal interpolation unit is configured to select linear interpolation, cubic spline interpolation, or similar day matching interpolation based on the missing span; The spatial interpolation reconstruction unit is configured to perform spatial reconstruction based on inverse distance weighting or bilinear interpolation. The dynamic anomaly detection module is configured to perform anomaly detection on the spatiotemporal aligned dataset based on sliding window dynamic statistical features and multi-dimensional context awareness mechanism, identify and label multiple anomaly types, and generate an anomaly label dataset. The intelligent repair and quality assessment module is configured to: calculate the comprehensive fit score of each candidate strategy in the repair strategy library based on the anomaly type, missing span, fluctuation degree, and spatial correlation or similarity of similar days, and select the strategy with the highest comprehensive fit score to repair the anomaly data in the anomaly-marked dataset; and calculate the comprehensive data quality score of the repaired data based on a quality index system bound to the sensitivity of photovoltaic power prediction errors; when the comprehensive data quality score is lower than a preset threshold, trigger a parameter callback mechanism to adjust the weight of the time base, the candidate data source set, the anomaly detection threshold, or the priority of the repair strategy, which includes: The repair strategy library management unit is configured to maintain a strategy library containing various repair strategies; The adaptive strategy selection unit is configured to calculate the strategy fit based on anomaly type, missing span, volatility, spatial correlation, or similarity of similar days and select the optimal strategy. The data quality assessment unit is configured to calculate a comprehensive quality score based on an indicator tied to the sensitivity of photovoltaic power prediction error. The parameter callback unit is configured to trigger adjustments to the time base weight, candidate source set, anomaly detection threshold, or policy priority when the quality score is below a threshold. The data traceability module is configured to record the entire lifecycle of data processing from input to output, generating data processing records that include processing operations, parameter configurations, and quality change trajectories for source analysis of data quality issues.

[0014] As a preferred embodiment of the multi-source heterogeneous data adaptive processing method for photovoltaic power prediction described in this invention, the parameter callback unit in the intelligent repair and quality assessment module is further configured to: when the quality score is lower than the threshold, adjust the weight coefficient of the time base selection, expand the candidate data source set, or adjust the priority order of each strategy in the repair strategy library according to the specific quality index that caused the low score.

[0015] As a preferred embodiment of the multi-source heterogeneous data adaptive processing method for photovoltaic power prediction described in this invention, the data tracing module includes: The processing log recording unit is configured to create and store a processing log for each data record using a chained structure, which includes the processing timestamp, processing module identifier, operation type, parameter configuration, and quality change amount. The data graph construction unit is configured to construct a directed acyclic graph of data based on the processing log to support forward tracing from the original data to the output data and reverse tracing from the output data to the original data. The quality problem tracing unit is configured to locate the root cause node of the quality problem based on the directed acyclic graph of the data when an abnormality in the output data is detected, and generate a tracing analysis report containing the problem propagation path and scope of impact.

[0016] Compared with existing technologies, the beneficial effects of this solution are: 1. This invention adopts an adaptive time reference selection mechanism, which dynamically evaluates and selects the optimal data source as the alignment reference based on multiple dimensions such as sampling frequency stability, data availability and cross-source latency consistency. This replaces the traditional fixed reference or simple resampling method, which can retain key time series information to the maximum extent, reduce artifacts and errors introduced by improper alignment, and significantly improve the quality of spatiotemporal aligned datasets.

[0017] 2. This invention introduces a multi-dimensional context-aware mechanism, including time, weather, and equipment status, to dynamically adjust the threshold for anomaly detection. This enables the system to intelligently distinguish between normal high fluctuations such as drastic weather changes and real data anomalies such as sensor malfunctions, effectively overcoming the problems of high false positive and false negative rates in traditional fixed threshold methods under non-stationary photovoltaic operating conditions.

[0018] 3. This invention constructs a strategy library containing multiple repair algorithms and designs an adaptive strategy selection model based on multi-dimensional features such as anomaly type, missing span, volatility, and spatiotemporal correlation. This model can match the optimal repair strategy for different types of anomalous data, avoiding the drawbacks of traditional single repair methods that are ineffective or introduce new errors, thus ensuring the accuracy and physical rationality of data repair.

[0019] 4. Furthermore, this invention can quantitatively evaluate data quality based on an indicator system linked to the sensitivity of photovoltaic prediction errors. When the quality fails to meet standards, it can automatically and purposefully adjust key parameters in preceding processing stages (such as time benchmark selection and anomaly detection), achieving continuous self-optimization of data processing quality. Simultaneously, the full lifecycle data traceability module makes the "black box" processing process transparent, providing a powerful tool for rapid location and root cause analysis of data quality problems, enhancing the system's maintainability and reliability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a system architecture diagram according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating multi-source heterogeneous data processing according to an embodiment of the present invention; Figure 3 This is a schematic diagram of context-aware anomaly detection according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the intelligent repair strategy selection process according to an embodiment of the present invention; Figure 5 This is a diagram illustrating the data quality assessment index system according to an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating data traceability according to an embodiment of the present invention; Figure 7 This is a flowchart illustrating the parameter callback process according to an embodiment of the present invention; Figure 8 A schematic diagram illustrating adaptive time base selection according to an embodiment of the present invention. Detailed Implementation

[0021] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0022] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0023] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0024] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0025] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0026] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0027] Example 1 Reference Figure 1 This is the first embodiment of the present invention, which provides a multi-source heterogeneous data adaptive processing system for photovoltaic power prediction. The system is deployed on a high-performance computing server or a cloud data processing platform, and interacts with data through a bus or microservice architecture. Figure 1 As shown, the system mainly includes the following five core modules: multi-source data access and standardization module, spatiotemporal alignment and interpolation reconstruction module, dynamic anomaly detection module, intelligent repair and quality assessment module, and data traceability module. The specific configuration and functions of each module are as follows: Specifically, for the multi-source data access and standardization module: this module is configured as the entry point of the data processing pipeline, responsible for shielding the differences in underlying hardware and protocols. Internally, it includes: Data Source Adaptation Unit: This unit has a built-in multi-protocol parsing engine configured to support Modbus, IEC 61850, OPC UA, MQTT, and HTTP / REST API protocols. For each type of data source (such as weather stations, inverters, and satellite cloud image servers), this unit is configured with an independent data parser to parse raw data packets and extract valid fields. Simultaneously, this unit maintains a data source metadata registry, recording the static characteristic parameters of each data source (sampling frequency, data precision, transmission latency, etc.).

[0028] Format standardization unit: Configured to perform standardization processing. Specifically, this includes: converting timestamps in different formats (such as Unix timestamps, non-standard strings) to the standard UTC time format, retaining precision to the millisecond level; and converting physical quantity units to the International System of Units (SI) (e.g., irradiance is standardized to SI). Temperature is standardized to °C, and power is standardized to kW; and weather type and other categorized data are uniformly encoded and mapped.

[0029] Data Stream Management Unit: It is configured to assign a globally unique identifier (UUID) to each data record entering the system, and to distribute standardized data to different cache queues according to data source type and time window, while monitoring the arrival rate and latency of the data stream in real time.

[0030] Specifically, for the spatiotemporal alignment and interpolation reconstruction module: this module is configured to map multi-source heterogeneous data to a unified spatiotemporal coordinate system. It internally comprises the following three core execution units: Adaptive Time Base Selection Unit: This unit does not use a fixed time base. Instead, it is configured to calculate three types of indicators from each data source and dynamically select the data source with the highest overall score as the time base source based on a weighted scoring mechanism. The calculation logic for the three types of indicators is as follows: Sampling frequency stability index S: reflects the stability of data source sampling. Its calculation formula is: in, The standard deviation of the sampling interval is represented by the standard deviation of the sampling interval. This represents the mean of the sampling interval.

[0031] It should be noted that the smaller the value of the standard deviation of the sampling interval divided by the mean of the sampling interval, the more stable the sampling frequency.

[0032] Data availability metric R: reflects the completeness of data from the data source. Its calculation formula is: in, This indicates the number of outliers and missing data points within the statistics window. This represents the total theoretical number of points within the statistics window.

[0033] Cross-source latency consistency metric C: reflects the time synchronization accuracy of the data source. Its calculation formula is: in, This represents the standard deviation of the data source timestamp from a reference clock (such as a GPS clock or Network Time Protocol NTP).

[0034] Furthermore, based on a weighted scoring mechanism, a comprehensive score is generated. The calculation is as follows: in, , , These are the weighting coefficients for each indicator. This unit selects... The highest data source is used as the benchmark for the current time window.

[0035] Hierarchical temporal interpolation unit: Configured to align data according to the target temporal resolution. For high-frequency data, downsampling aggregation (mean / extreme values) is used; for low-frequency data, linear interpolation (short time), cubic spline interpolation (medium duration), or similar day matching interpolation (long time) algorithms are called hierarchically based on the missing span.

[0036] Spatial interpolation and reconstruction unit: configured to handle data alignment in the spatial dimension. For discrete measurement points, a spatial topology model is established and reconstructed using the inverse distance weighted method (IDW); for raster data such as satellite cloud images, we use bilinear interpolation to unify spatial resolution.

[0037] Specifically, the bilinear interpolation calculation formula is as follows: in, The values ​​of the points to be interpolated; , , , The values ​​are the values ​​of the four grid points surrounding the point to be interpolated (usually, interpolation is performed first in the X direction to obtain intermediate results, and then interpolation is performed in the Y direction). , , , This represents the distance weight between the point to be inserted and each grid point along the corresponding coordinate axis.

[0038] Specifically, the dynamic anomaly detection module is configured to identify anomalous patterns in the spatiotemporally aligned dataset. Internally, it includes: The sliding window statistical characteristic calculation unit is configured to calculate the mean, standard deviation, skewness, kurtosis and other statistics of data within multi-scale windows (real-time, ultra-short-term, short-term, etc.) and establish a dynamic baseline.

[0039] Context-aware anomaly detection unit: This unit introduces multi-dimensional contextual features (time, weather, equipment status) to dynamically adjust the anomaly detection threshold.

[0040] Specifically, the dynamically adjusted anomaly detection threshold can be expressed as: in, The adjusted dynamic threshold; Based on the threshold (such as the range of irradiance fluctuation under sunny conditions) ); Context adjustment factor; The weights of the i-th class of context features; This is the quantized value of the i-th type of contextual feature (such as cloud cover level, device status code).

[0041] Anomaly classification and labeling unit: It is configured to identify anomalies as missing, out-of-bounds, mutated, drifting, or frozen based on a multi-rule joint decision mechanism, and write the anomaly information to the anomaly log table (data_anomaly_log).

[0042] Specifically, the intelligent repair and quality assessment module (including parameter callback functionality) is the core decision-making and feedback center of the system, and includes: Repair Strategy Library Management Unit: Configured to maintain the strategy library, it has built-in multiple algorithms such as linear interpolation, spline interpolation, similar day replacement, physical constraints, neighbor point weighting, and exponential smoothing.

[0043] The adaptive strategy selection unit is configured to calculate the fitness score of each candidate strategy and select the best one to execute. The fitness score is calculated using the following formula: in, , , , , where represents the weighting coefficients for each dimension; L represents the matching degree between the missing time span and the strategy; V represents the matching degree between the fluctuation characteristics before and after the anomaly and the strategy; P represents the matching degree between the anomaly type and the strategy; and D represents the matching degree between similar days or spatially adjacent points and the strategy. This is the normalization function.

[0044] Data Quality Assessment Unit: Configured to calculate the overall quality score of the repaired data, with its indicator system linked to the sensitivity of photovoltaic power prediction errors. This data quality assessment unit also includes the following indicators: Timing alignment error index : Where N is the number of samples, For the aligned timestamps, This is the original sampling timestamp. This is the standard sampling period.

[0045] Irradiation-Power Consistency Residual Index : in, This is the measured power. Based on the total horizontal irradiance (GHI) and conversion efficiency The theoretical power calculated from the photovoltaic array area A; This refers to the rated power of the power station.

[0046] Prediction window availability index : in, To predict the number of valid data points within the window after repair, The number of theoretical data points required for the prediction model.

[0047] Parameter callback unit: Configured to execute closed-loop feedback. This occurs when the overall quality score Q calculated using the analytic hierarchy process is below a preset threshold. At that time, the unit automatically triggers an adjustment command: like If the score is low, increase the weight of the S-index in the time benchmark selection. ; like If the score is low, the set of candidate sources for the time benchmark is expanded; If the repair is ineffective, adjust the priority of each strategy in the repair strategy library.

[0048] Specifically, the data traceability module is configured as a "black box" of the data processing process, and its internal components include: Processing log recording unit: It is configured to store information of each processing node (input characteristics, operation type, parameter configuration, output characteristics) in a chain structure and write it to the data processing log table.

[0049] Data graph construction unit: It is configured to build a directed acyclic graph (DAG) based on logs, where nodes represent data states and edges represent processing operations.

[0050] Quality issue tracing unit: It is configured to perform a reverse search on the DAG graph when the output data quality is abnormal, locate the root cause node that caused the quality degradation (such as abnormal raw data acquisition or improper setting of specific interpolation parameters), and generate a tracing report.

[0051] Example 2 Reference Figures 2 to 8 This is the second embodiment of the present invention, which provides an adaptive processing method for multi-source heterogeneous data for photovoltaic power prediction. This embodiment takes a photovoltaic power plant cluster data processing scenario as an example, which includes 18 sites (siteid 1~18). The focus is on processing the missing historical meteorological data (eval_measure_weather table) and predicted power data (eval_forecast_power table) for siteid 18 from July 1, 2013 to January 20, 2016. The method includes the following steps: S1: Multi-source data access and standardization. For example... Figure 3 As shown, data access is performed first. The steps are as follows: S1.1 Data Source Adaptation: The system accesses real-time data from field weather stations via the Modbus TCP protocol and accesses third-party numerical weather prediction (NWP) data via an HTTP REST API. For cases where siteid18 lacks data, the system uses historical archived data (stored in a MySQL database) from the same area's siteid2 station as a supplementary source.

[0052] S1.2 Format Standardization: The timestamp strings in the original siteid2 data (e.g., "2013 / 07 / 01 12:00") are uniformly converted to the UTC standard time format ("2013-07-01 12:00:00"). The irradiance unit is standardized to... Temperatures are uniformly set to °C. Weather types are encoded and mapped (e.g., sunny = 1, cloudy = 2).

[0053] S1.3 Data Flow Management: Assign a globally unique identifier (UUID) to each access record, such as DATA_20130701_1200_001, and store it in a standardized cache queue.

[0054] S2: Adaptive time base selection and spatiotemporal alignment. For example... Figure 8 As shown, to align the data of siteid2 to the timeline of siteid18, the following adaptive time base selection is performed: S2.1 Calculate scoring indicators: For candidate data sources (such as measured weather tables, NWP tables), calculate three types of indicators.

[0055] Specifically, the three categories of indicators are sampling frequency stability, data availability, and cross-source latency consistency.

[0056] S2.2 Comprehensive Score Selection: Calculate the comprehensive score based on the weighted scoring mechanism.

[0057] S2.3 Layered Time Interpolation: For data with short-term missing intervals (≤2 sampling periods), linear interpolation is used.

[0058] For moderate missing values ​​(3-6 sampling periods), cubic spline interpolation is used.

[0059] For data that has been missing for a long period of time (>6 sampling periods), such as siteid18 in this example which has been missing for 22 months, it is marked as to be repaired and proceeds to step S4 for processing.

[0060] S2.4 Spatial Interpolation Reconstruction: For satellite cloud raster data, bilinear interpolation is used to unify spatial resolution.

[0061] S3: Context-aware dynamic anomaly detection, such as Figure 3 and Figure 4 As shown, anomaly detection is performed on the aligned data.

[0062] S3.1 Sliding Window Statistics: Set a real-time window (15 minutes), calculate statistical characteristics such as mean and standard deviation, and update the dynamic baseline using exponentially weighted moving average (EWMA).

[0063] S3.2 Context Threshold Adjustment: Obtain contextual features: time (summer noon), weather (cloudy), and equipment (normal operation).

[0064] Calculate dynamic context thresholds.

[0065] S3.3 Anomaly Marker: If the irradiance of a data point (e.g., 1600 ppm at 12:00:00 on 2014-06-15) is 1600 ppm... If both the "threshold exceeded" and "rate of change abnormal" rules are triggered simultaneously, it is judged as an "exceeding limit anomaly" and written to the data_anomaly_log table.

[0066] S4: Intelligent policy library for optimal repair. For example... Figure 5 As shown, intelligent repair is performed for the anomalies detected in step S3 and the long-term missing data left over from step S2.

[0067] S4.1 Fit Calculation: For the 22 months of meteorological data missing in siteid18, the system calculates the fit score for each strategy.

[0068] S4.2 Similar Date Repair Execution: The system selects the "Similar Date Replacement Repair" strategy.

[0069] Feature extraction: Extract features such as season and weather type of the date to be filled.

[0070] Similar day search: Search the siteid2 historical database for the top-3 similar days (e.g., 2015-06-14, 2015-06-16, 2015-06-20) and calculate the similarity weights (0.35, 0.33, 0.32).

[0071] Weighted fusion: Execute the SQL insert operation to generate supplementary data.

[0072] S5: Quality assessment and parameter callback. For example... Figure 5 and Figure 8 As shown, a closed-loop evaluation is performed on the repaired data.

[0073] S5.1 Quality Indicator Calculation. This includes calculating the timing alignment error, irradiance-power consistency residual, and in-prediction availability.

[0074] S5.2 Parameter Callback: The overall score is calculated using the analytic hierarchy process.

[0075] If the overall score is less than the threshold of the overall score and the timing alignment error index contributes the most, then the sampling frequency stability weight in step S2 will be automatically increased.

[0076] If the overall score is less than the threshold of the overall score and the contribution of the irradiance-power consistency residual index is the largest, then the candidate data source set is expanded.

[0077] If the repair is ineffective, increase the priority of "Similar Day Replacement" or "Physical Constraint Repair" in the strategy library.

[0078] S6: Data lifecycle traceability. For example... Figure 6 and Figure 7 As shown, the entire data processing process is recorded.

[0079] Log recording: Record each operation (such as "linear interpolation", "similar day supplementation") and its input and output quality in the data_processing_log table.

[0080] Source tracing analysis: When the output data quality is abnormal (e.g., overall_score < 0.8), construct a DAG graph for reverse search, locate the root cause node (e.g., "original data acquisition sensor failure"), and generate a source tracing report containing the problem propagation path.

[0081] In addition, this embodiment provides the table structures for data_anomaly_log and data_processing_log, as well as the corresponding execution code.

[0082] Specifically, the structure of the data_anomaly_log table and its execution code are as follows: CREATE TABLE `data_anomaly_log` ( `id` BIGINT NOT NULL AUTO_INCREMENT, `anomaly_id` VARCHAR(50) NOT NULL COMMENT 'Unique identifier for anomalies (UUID)', `station_id` VARCHAR(50) NOT NULL COMMENT 'Station ID (e.g., siteid18)', `data_table` VARCHAR(100) NOT NULL COMMENT 'The table containing the abnormal data (e.g., eval_measure_weather)', `data_id` BIGINT NOT NULL COMMENT 'The primary key ID of the abnormal data in the original table', `curtime` DATETIME NOT NULL COMMENT 'Abnormal data timestamp', --Abnormal type and severity `anomaly_type` TINYINT NOT NULL COMMENT 'Anomaly type: 1. Missing, 2. Out of bounds, 3. Mutation, 4. Drift, 5. Freeze', `anomaly_severity` TINYINT NOT NULL COMMENT 'Severity level: 1 Mild, 2 Moderate, 3 Severe', `anomaly_value` DECIMAL(20,6) NULL COMMENT 'Outlier (out-of-limit / mutation, etc.)', `expected_range` VARCHAR(100) NULL COMMENT 'Expected reasonable range (e.g., "0-1500")', --Testing basis `detection_rules` VARCHAR(255) NOT NULL COMMENT 'The triggered exception rules (such as "threshold exceeded, abnormal rate of change")', `context_info` JSON NOT NULL COMMENT 'Context information (time, weather, equipment status, etc.)', `confidence` DECIMAL(5,4) NOT NULL COMMENT 'Anomaly detection confidence (0-1)', --Repair status `repair_status` TINYINT DEFAULT 0 COMMENT 'Repair status: 0 Not repaired, 1 Repaired, 2 Repair failed', `repair_method` VARCHAR(50) NULL COMMENT 'Repair method (e.g., "similar date replacement")', `repair_time` DATETIME NULL COMMENT 'Repair time', `repair_confidence` DECIMAL(5,4) NULL COMMENT 'Repair confidence', `create_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'Record creation time', `update_time` DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT 'update time', PRIMARY KEY (`id`), UNIQUE KEY `uk_anomaly_id` (`anomaly_id`), KEY `idx_station_time` (`station_id`, `curtime`), KEY `idx_anomaly_type` (`anomaly_type`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Data Anomaly Log Table'; Specifically, the structure of the data_processing_log table and its execution code are as follows: CREATE TABLE `data_processing_log` ( `id` BIGINT NOT NULL AUTO_INCREMENT, `log_id` VARCHAR(50) NOT NULL COMMENT 'Unique Log Identifier (UUID)', `data_uuid` VARCHAR(50) NOT NULL COMMENT 'Globally unique identifier for the data (associated with the original data)', `station_id` VARCHAR(50) NOT NULL COMMENT 'station ID', `data_table` VARCHAR(100) NOT NULL COMMENT 'data_table_name', `data_id` BIGINT NOT NULL COMMENT 'The primary key ID of the data in the original table', --Process node information `processing_module` VARCHAR(50) NOT NULL COMMENT 'Processing module (e.g., "anomaly detection")', `operation_type` VARCHAR(50) NOT NULL COMMENT 'Operation type (e.g., "linear interpolation")', `operation_params` JSON NULL COMMENT 'Operation parameters (e.g., {"window_size":5})', `input_data` JSON NULL COMMENT 'Input data (variable part)', `output_data` JSON NULL COMMENT 'Output data (changed parts)', --Mass Change `input_quality` DECIMAL(5,4) NULL COMMENT 'Quality score before processing', `output_quality` DECIMAL(5,4) NULL COMMENT 'Processed quality score', `quality_change` DECIMAL(5,4) NULL COMMENT 'Change in quality', `create_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'Processing time', PRIMARY KEY (`id`), UNIQUE KEY `uk_log_id` (`log_id`), KEY `idx_data_uuid` (`data_uuid`), KEY `idx_station_time` (`station_id`, `create_time`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Data Processing Log Table'; Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0083] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0084] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0085] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0086] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0087] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An adaptive processing method for multi-source heterogeneous data for photovoltaic power prediction, characterized in that, include: S1: Access raw data from multiple data sources, and perform format conversion, unit unification, and encoding standardization on the raw data to generate a standardized data stream; S2: Calculate the sampling frequency stability, data availability, and cross-source latency consistency index of each data source, and select the data source with the highest comprehensive score as the time reference source based on the weighted scoring mechanism. Perform time alignment on the standardized data stream according to the target time resolution, and perform spatial interpolation reconstruction based on spatial correlation to generate a spatiotemporally aligned dataset. S3: Based on the sliding window dynamic statistical features and multi-dimensional context awareness mechanism, perform anomaly detection on the spatiotemporal aligned dataset, identify and label multiple anomaly types, and generate an anomaly labeling dataset; S4: Calculate the comprehensive fit score of each candidate strategy in the repair strategy library based on the anomaly type, missing span, fluctuation degree and spatial correlation or similarity of similar days, and select the strategy with the highest comprehensive fit score to perform repair on the abnormal data in the anomaly labeled dataset. S5: Calculate the comprehensive data quality score of the repaired data based on the quality index system bound to the sensitivity of photovoltaic power prediction error; When the overall data quality score is lower than a preset threshold, a parameter callback mechanism is triggered to adjust the weight of the time base, the candidate data source set, the anomaly detection threshold, or the priority of the repair strategy. S6: Records the entire lifecycle of data processing from input to output, generating data processing records that include processing operations, parameter configurations, and quality change trajectories for source tracing analysis of data quality issues.

2. The adaptive processing method for multi-source heterogeneous data for photovoltaic power prediction as described in claim 1, characterized in that, The data source access and standardization in S1 include: By adapting to and parsing raw data packets from various data sources using multiple data access protocols, valid data fields are extracted. Establish a data source metadata registry to record characteristic parameters such as sampling frequency and data precision of each data source; Convert timestamps of different formats to the Coordinated Universal Time (UTC) standard time format; Convert physical quantities of different units into the International System of Units (SI); Assign a globally unique identifier to each data record and cache standardized data in separate streams according to data source type and time window.

3. The adaptive processing method for multi-source heterogeneous data for photovoltaic power prediction as described in claim 1, characterized in that, The time alignment and spatial interpolation reconstruction in S2 include: For data with a sampling frequency higher than the target time resolution, downsampling aggregation is used; For data with a sampling frequency lower than the target time resolution, a hierarchical interpolation strategy is adopted based on the time span of missing data, and linear interpolation, cubic spline interpolation, or similar day matching interpolation are selected for processing. Based on the spatial topological relationship of each measuring point, the inverse distance weighting method is used to interpolate the spatially sparse points; For raster data, bilinear interpolation is used to unify spatial resolution.

4. The adaptive processing method for multi-source heterogeneous data for photovoltaic power prediction as described in claim 1, characterized in that, The anomaly detection in S3 includes: Calculate the statistical characteristics of the data within a multi-scale sliding window and establish a dynamic baseline for the statistical characteristics; Construct a multi-dimensional contextual feature vector that includes time context, meteorological context, and equipment context; The threshold used for anomaly detection is dynamically adjusted based on the context feature vector. A multi-rule joint judgment mechanism is adopted. When data triggers multiple abnormal rules at the same time, the abnormality type is comprehensively determined, and the detected abnormalities are marked as missing, out-of-limit, mutation, drift, or frozen.

5. The adaptive processing method for multi-source heterogeneous data for photovoltaic power prediction as described in claim 1, characterized in that, The context-aware anomaly detection in S3 includes: Obtain the time context features of the current data point, including time, date type, and season; Obtain the meteorological context features of the current data point, including weather type and cloud cover level; Obtain the device context features of the current data point, including device operating status and historical fault records; Based on the aforementioned time context features, meteorological context features, and equipment context features, the anomaly detection threshold is adjusted together, and a multi-rule joint decision is executed to output the anomaly detection result.

6. The adaptive processing method for multi-source heterogeneous data for photovoltaic power prediction as described in claim 1, characterized in that, The repair strategy library in S4 includes the following strategies: Linear interpolation repair, spline interpolation repair, similar day replacement repair, physical constraint repair, nearest neighbor point weighted repair, and exponential smoothing repair; Each repair strategy is configured with its applicable conditions, parameter range, and priority.

7. The adaptive processing method for multi-source heterogeneous data for photovoltaic power prediction as described in claim 1, characterized in that, The quality score evaluation and parameter callback in S5 include: The timing alignment error index, irradiance-power consistency residual index, and prediction window availability index were calculated, and the comprehensive data quality score was calculated using the analytic hierarchy process. The weight of each index was determined based on its impact on the accuracy of photovoltaic power prediction. When the overall data quality score is lower than the preset threshold and the time alignment error index contributes the most to the low score, increase the weight of the sampling frequency stability index in the time reference selection. When the overall data quality score is lower than the preset threshold and the irradiance-power consistency residual index contributes the most to the low score, the set of candidate data sources for the expanded time base is selected. If the data quality still does not meet the requirements after repair, adjust the priority order of each strategy in the repair strategy library.

8. A multi-source heterogeneous data adaptive processing system for photovoltaic power prediction, characterized in that, include: The multi-source data access and standardization module is configured to: access raw data from multiple data sources, and perform format conversion, unit unification, and encoding standardization on the raw data to generate a standardized data stream; The spatiotemporal alignment and interpolation reconstruction module is configured to perform the following: calculate the sampling frequency stability, data availability, and cross-source latency consistency indicators of each data source; select the data source with the highest comprehensive score as the time reference source based on a weighted scoring mechanism; perform time alignment of the standardized data stream according to the target time resolution; perform spatial interpolation reconstruction based on spatial correlation; and generate a spatiotemporal aligned dataset, which includes: The adaptive time reference selection unit is configured to calculate and select a time reference source based on three types of indicators: sampling frequency stability, data availability, and cross-source latency consistency. The hierarchical temporal interpolation unit is configured to select linear interpolation, cubic spline interpolation, or similar day matching interpolation based on the missing span; The spatial interpolation reconstruction unit is configured to perform spatial reconstruction based on inverse distance weighting or bilinear interpolation. The dynamic anomaly detection module is configured to perform anomaly detection on the spatiotemporal aligned dataset based on sliding window dynamic statistical features and multi-dimensional context awareness mechanism, identify and label multiple anomaly types, and generate an anomaly label dataset. The intelligent repair and quality assessment module is configured to: calculate the comprehensive fit score of each candidate strategy in the repair strategy library based on the anomaly type, missing span, fluctuation degree, and spatial correlation or similarity of similar days, and select the strategy with the highest comprehensive fit score to repair the anomaly data in the anomaly-marked dataset; and calculate the comprehensive data quality score of the repaired data based on a quality index system bound to the sensitivity of photovoltaic power prediction errors; when the comprehensive data quality score is lower than a preset threshold, trigger a parameter callback mechanism to adjust the weight of the time base, the candidate data source set, the anomaly detection threshold, or the priority of the repair strategy, which includes: The repair strategy library management unit is configured to maintain a strategy library containing various repair strategies; The adaptive strategy selection unit is configured to calculate the strategy fit based on anomaly type, missing span, volatility, spatial correlation, or similarity of similar days and select the optimal strategy. The data quality assessment unit is configured to calculate a comprehensive quality score based on an indicator tied to the sensitivity of photovoltaic power prediction error. The parameter callback unit is configured to trigger adjustments to the time base weight, candidate source set, anomaly detection threshold, or policy priority when the quality score is below a threshold. The data traceability module is configured to record the entire lifecycle of data processing from input to output, generating data processing records that include processing operations, parameter configurations, and quality change trajectories for source analysis of data quality issues.

9. The multi-source heterogeneous data adaptive processing system for photovoltaic power prediction as described in claim 8, characterized in that, The parameter callback unit in the intelligent repair and quality assessment module is also configured to: when the quality score is lower than the threshold, adjust the weight coefficient of the time base selection, expand the candidate data source set, or adjust the priority order of each strategy in the repair strategy library according to the specific quality indicator that caused the low score.

10. The multi-source heterogeneous data adaptive processing system for photovoltaic power prediction as described in claim 8, characterized in that, The data traceability module includes: The processing log recording unit is configured to create and store a processing log for each data record using a chained structure, which includes the processing timestamp, processing module identifier, operation type, parameter configuration, and quality change amount. The data graph construction unit is configured to construct a directed acyclic graph of data based on the processing log to support forward tracing from the original data to the output data and reverse tracing from the output data to the original data. The quality problem tracing unit is configured to locate the root cause node of the quality problem based on the directed acyclic graph of the data when an abnormality in the output data is detected, and generate a tracing analysis report containing the problem propagation path and scope of impact.

Images