Power monitoring data missing filling method combining HDBSCAN and multi-model mechanism

Abstract

This invention relates to the field of missing data imputation technology, and discloses a method for imputing missing power monitoring data by combining HDBSCAN and a multi-model mechanism. First, the transformer area data is divided into Class I and Class II data according to the photovoltaic operating status recorded by the SCU. Then, within each class, the HDBSCAN algorithm is used to further divide the data into Class II data, automatically forming cluster structures based on local density differences in sample distribution and automatically identifying noise points. Finally, an adaptive structure switching mechanism based on missing data length is constructed. Depending on the length of the missing interval in the transformer area data and the operating condition type, LSTM, Attention-LSTM, or BiLSTM-Attn models are dynamically selected for training, providing targeted modeling for different operating condition characteristics and missing data spans, thus achieving the prediction and imputation of missing data. This invention achieves accurate characterization of data distribution under different operating conditions, enhances the model's adaptability to complex missing patterns, and effectively improves the accuracy of data imputation.

Description

Technical Field

[0001] This invention relates to the field of missing data imputation technology, specifically a method for imputing missing power monitoring data that combines HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) with a multi-model mechanism. Background Technology

[0002] With the large-scale integration of distributed photovoltaic (PV) power generation into the distribution network, power quality issues such as voltage exceeding limits, harmonic pollution, and three-phase imbalance are becoming increasingly prominent. Voltage, current, and other power monitoring data are prerequisites for power quality analysis and management in the distribution network; therefore, real-time and accurate monitoring is crucial. As a key link connecting the distribution network and end-users, distribution transformer substations have seen widespread adoption in recent years to achieve comprehensive perception and real-time tracking of their operational status. These substations can acquire and analyze power data in real time, exhibiting high flexibility and responsiveness. The SCU is an intelligent data acquisition and uploading device used in distribution network substations, possessing data fusion, edge computing, and real-time transmission capabilities, enabling real-time monitoring of electrical quantities such as voltage, current, and power. However, in real-world applications, changes in the collector's operating environment, network fluctuations during data upload, equipment malfunctions, and human error can all affect the accuracy of SCU data acquisition. Therefore, data loss inevitably occurs during actual SCU data uploads, severely impacting data quality and hindering subsequent analysis and evaluation of power quality on the user side. Therefore, accurately filling in the missing data in the SCU is one of the important prerequisites for ensuring the quality of power supply in the distribution area.

[0003] To address the problem of missing data, researchers have proposed several methods for imputing missing data. The first category is simple imputation methods, including mean imputation, median imputation, forward imputation, and backward imputation. These methods are easy to use, but they only utilize local information and cannot capture the relationship between missing data and the global data. When the missing data ratio is high or the data structure is complex, the imputation accuracy is low. The second category is interpolation-based imputation methods, including linear interpolation, polynomial interpolation, and spline function interpolation. These methods consider the trend of data transformation to some extent and have good imputation effects when dealing with small-scale, regularly distributed missing data. However, when dealing with large-scale missing data, nonlinear or non-stationary datasets, they may exhibit underfitting or poor stability, resulting in less than ideal imputation results. Therefore, researchers have used a variety of complementary methods to improve the accuracy and stability of data imputation. These methods effectively improve the accuracy and stability of data imputation and achieve good results in low-dimensional, stationary data scenarios with relatively regular missing data patterns.

[0004] However, the large-scale integration of photovoltaic equipment has significantly increased the complexity of the data in the distribution area, posing new challenges to the accuracy and stability of data filling methods, mainly in the following three aspects:

[0005] (1) High-dimensional characteristics: The monitoring data of the transformer area covers a variety of physical quantities such as voltage, current and power of transformer and photovoltaic equipment. There are complex time-series dependencies between variables. Existing filling methods take into account the correlation between variables when processing high-dimensional data, which makes it difficult to maintain physical consistency of the filling results, thus affecting the filling accuracy and generalization ability.

[0006] (2) Non-stationarity: Affected by factors such as photovoltaic power output fluctuations and load changes, the data of the distribution area shows obvious time-series variation characteristics, making it difficult for filling methods based on static statistical assumptions or fixed patterns to be accurately adapted.

[0007] (3) Complex missing data patterns: Data from photovoltaic power distribution areas may contain random missing data or large-scale continuous missing data. Current methods perform well in handling random missing data, but when faced with large-scale continuous missing data, interpolation methods are prone to error accumulation, while machine learning-based imputation methods may significantly reduce prediction accuracy due to the long span of missing data. Therefore, there is an urgent need for an efficient data imputation method that adapts to the complex data characteristics of photovoltaic power distribution areas.

[0008] Therefore, in the operation monitoring and analysis of distribution networks containing photovoltaics, the data uploaded by SCU and transformer substations presents the following technical challenges:

[0009] (1) Data loss is common and complex. Due to unstable communication, equipment accuracy and network interruption, key measurement data such as voltage, current and power are often continuously or intermittently lost. The length of the missing interval varies greatly, ranging from short missing points to long whole segments, making it difficult for a single model to take into account all the data.

[0010] (2) With diverse and mixed operating conditions, the distribution area shows significant differences between the "photovoltaic operation period" and the "outage period" under the background of photovoltaic access. Under the same operating condition, there may also be multiple fine-grained operating condition modes (load level, power factor, etc.). If modeling is carried out directly without distinction, the distribution is very likely to be mixed, which will lead to a decrease in prediction accuracy.

[0011] (3) Limitations of existing methods. Traditional interpolation or single deep learning models are often only applicable to fixed missing patterns and cannot cope with multiple working conditions and multiple spans of missing data. Existing clustering methods mostly rely on manually setting the number of clusters, lack the ability to process noise points, and are difficult to stably identify the actual working condition structure. There is a lack of adaptive mechanism, and the model complexity cannot be dynamically adjusted under different missing lengths, which may lead to a waste of computing resources or inaccurate predictions due to insufficient models. Summary of the Invention

[0012] To address the aforementioned problems, the present invention aims to provide a power monitoring data missing data imputation method that combines HDBSCAN with a multi-model mechanism. This method achieves accurate characterization of data distribution under different operating conditions, enhances the model's adaptability to complex missing patterns, effectively improves data imputation accuracy, and solves the problem that traditional methods struggle to adapt to complex missing patterns. Furthermore, by specifically modeling the characteristics and missing data spans of different operating conditions, it avoids error accumulation problems under conditions of large-scale continuous or non-uniform missing data, significantly enhancing the model's ability to capture long-term dependencies and the stability of the imputation results. The technical solution is as follows:

[0013] A method for filling missing data in power monitoring data that combines HDBSCAN with a multi-model mechanism includes the following steps:

[0014] Step 1: Divide the photovoltaic operating status data of the transformer substation into primary categories according to the photovoltaic operating status recorded by the intelligent integrated terminal: If the photovoltaic equipment is in operation, it enters the Class I time period, which includes both photovoltaic data and transformer data; if the photovoltaic equipment is not in operation, it enters the Class II time period, which includes only transformer data.

[0015] Step 2: Within each data category, the HDBSCAN algorithm is used to perform secondary partitioning of the data. Based on the local density differences in the sample distribution, cluster structures are automatically formed, and noise points are automatically identified to achieve a refined division of operating conditions.

[0016] Step 3: Construct an adaptive structure switching mechanism based on missing data length. According to the length of the missing interval in the transformer area data and the operating condition type, dynamically select LSTM, Attention-LSTM or BiLSTM-Attn model for training, and model the characteristics of different operating conditions and missing data spans in a targeted manner to achieve prediction and filling of missing data.

[0017] The beneficial effects of this invention are:

[0018] (1) This invention proposes a hierarchical data preprocessing method based on photovoltaic operating status and HDBSCAN operating condition division. By first dividing the data according to photovoltaic operating status (OpSt), and then using the HDBSCAN algorithm to perform secondary clustering within each data category, the method automatically identifies homogeneous operating condition subsets and removes noise points, avoiding the distribution shift problem caused by heterogeneous data mixing modeling. This ensures that model training and inference are carried out under a consistent distribution, thereby significantly improving the accuracy and stability of missing data imputation. At the same time, HDBSCAN has the advantages of no cluster number setting and stable cluster selection, avoiding the uncertainty of manual parameter tuning and ensuring the robustness and transferability of clustering results.

[0019] (2) This invention proposes a multi-model imputation mechanism based on adaptive switching of missing interval length. By statistically analyzing the distribution of missing interval lengths, thresholds L1 and L2 are adaptively determined using quantiles, enabling dynamic selection of LSTM / Attention-LSTM / BiLSTM-Attn models: small-span missing intervals are efficiently recovered by LSTM to restore local temporal sequence; medium-span missing intervals are reinforced by introducing an attention mechanism to enhance key moment memory; and long-span missing intervals are captured by using the bidirectional dependency of BiLSTM combined with an attention mechanism to capture the global structure. This mechanism matches the "model memory range" with the "missing time scale," which reduces the overall error and suppresses error accumulation, while improving computational efficiency and interpretability while ensuring imputation accuracy. Attached Figure Description

[0020] Figure 1 This is a flowchart of the power monitoring data missing filling method that combines HDBSCAN and a multi-model mechanism according to the present invention.

[0021] Figure 2 This is a comparison chart of the actual filling effect of the present invention with that of mainstream methods. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0023] This invention proposes a missing data imputation method for power monitoring combining HDBSCAN and a multi-model mechanism. First, a data clustering method based on photovoltaic (PV) operating conditions is proposed. Through initial data partitioning based on PV equipment operating status (OpSt) and further subdivision of the data clusters based on HDBSCAN clustering, accurate characterization of data distribution under different operating conditions is achieved, enhancing the model's adaptability to complex missing patterns and effectively improving data imputation accuracy, thus solving the problem that traditional methods struggle to adapt to complex missing patterns. Then, based on this, this invention further proposes an adaptive multi-model LSTM imputation mechanism: dynamically selecting LSTM, Attention-LSTM, or BiLSTM-Attn models for training and prediction based on the length of the missing interval and the operating condition type, thereby achieving targeted modeling of different operating condition characteristics and missing spans. This mechanism not only avoids the error accumulation problem of traditional LSTM under large-scale continuous or non-uniform missing conditions but also significantly enhances the model's ability to capture long-term dependencies and the stability of the imputation results. The flowchart is as follows. Figure 1 As shown. The specific implementation process is as follows:

[0024] 1. First-level classification of data based on photovoltaic operating status.

[0025] The operating status of photovoltaic (PV) equipment can serve as an important basis for dividing the dataset. This invention first classifies the data according to the PV operating status (OpSt, Operation State of PV) recorded by the SCU (Smart Converged Terminal) to reduce the impact of mixed data patterns on the model completion.

[0026] The specific divisions are as follows:

[0027] (1) When OpSt = 1, the photovoltaic equipment is in working condition and enters the Class I time period (photovoltaic working period). At this time, Class I data includes photovoltaic data and transformer data.

[0028] (2) When OpSt = 0, the photovoltaic equipment stops working and enters the Class II time period (photovoltaic shutdown period). At this time, the Class II data only includes transformer data.

[0029] This hierarchical approach ensures the overall consistency of the dataset, providing a foundation for subsequent data stacking.

[0030] 2. Secondary partitioning of data based on HDBSCAN clustering.

[0031] After initially classifying the photovoltaic operating states, this invention further employs the HDBSCAN algorithm for secondary classification within each data category. This method automatically forms cluster structures based on local density differences in sample distribution and automatically identifies noise points, thereby achieving refined classification of operating conditions and ensuring that subsequent model training is conducted on a homogeneous dataset.

[0032] HDBSCAN is a density-based hierarchical clustering algorithm that automatically identifies cluster structures formed by high-density regions by constructing density maps and minimum spanning trees (MSTs), and automatically removes noise points, avoiding the need for manual setting of the number of clusters.

[0033] HDBSCAN clustering involves the following three steps in its secondary partitioning of data:

[0034] Step 2.1: Construct the density map and minimum spanning tree (MST);

[0035] A minimum spanning tree (MST) is, in graph theory, the set of edges connecting all nodes with the minimum sum of edge weights. In this invention, the MST is constructed based on the mutual reachability distance of sample points to reflect the density structure of the operating data.

[0036] In the input dataset, the key parameters of HDBSCAN are first determined:

[0037] (1) Minimum number of samples min_samples: Usually, the value is 2 to 3 times the feature dimension. For example, when the input features include four key quantities such as voltage, current, active power and reactive power, the corresponding feature dimension is d=4. Then, min_samples=8-12 can be selected to ensure that a minimum stable working condition unit can be covered when clustering the working conditions, thereby ensuring the reliability of local density estimation.

[0038] Minimum Samples is one of the core parameters in the HDBSCAN algorithm. It defines the minimum number of neighborhood samples required to form a density core, and is usually taken as 2 to 3 times the feature dimension.

[0039] (2) Minimum cluster size min_cluster_size: Using the statistical proportion method, a fixed proportion of the total sample size N is taken, generally 5%, i.e.:

[0040] (1);

[0041] This method ensures that the resulting clusters are large enough to be representative.

[0042] Minimum Cluster Size is also one of the core parameters of HDBSCAN. It defines the minimum number of samples required for cluster formation. It is usually taken as a fixed percentage of the total number of samples (such as 5%) to ensure the representativeness of the cluster.

[0043] Based on this, the core distance (core_dist) and mutual reach distance (mreach_dist) of each sample point are calculated:

[0044] (2);

[0045] (3);

[0046] in, Let u be the feature vector of the u-th sample; and Representing sample points respectively and The Euclidean distance to its k-th nearest neighbor; Represents sample points and The Euclidean distance between them.

[0047] Core distance is the Euclidean distance from a sample point to its k-th nearest neighbor, used to characterize the local density level of the point. Mutual reachability distance is defined as the maximum of the distance between two sample points and their core distance, used to construct the weighted graph and MST.

[0048] A weighted graph is constructed based on mutual reachability distance, and Kruskal's algorithm is used to generate a minimum spanning tree (MST). In this invention, the optimization objective of the MST is:

[0049] (4);

[0050] Where G represents a weighted graph consisting of sample points, and Tr represents a candidate spanning tree, which is a set of edges in graph G that connect all sample points without forming loops. The weights of the edges (i.e., the sample points) and (Mutual reachability distance). The MST obtained through this optimization process can ensure that points in dense regions are connected first, while sparse regions or outliers are weakened, laying the foundation for the generation of subsequent density hierarchy trees.

[0051] Step 2.2: Construct a density hierarchy tree and extract stable clusters;

[0052] Based on the generated MST, HDBSCAN constructs a density hierarchy tree (Condensed Tree) by progressively increasing the distance threshold and automatically filters effective clusters using cluster stability scores. In this invention, noise points refer to sample points that cannot form an effective density core within the min_samples neighborhood. These points typically temporarily belong to a cluster at low density levels but are quickly eliminated at higher densities, thus failing to persist long-term. HDBSCAN marks these points as outliers (cluster label -1) to prevent them from interfering with subsequent model training.

[0053] To screen stable structures from candidate clusters, this invention introduces a cluster stability score, defined as:

[0054] (5);

[0055] in, For clusters The stability score is used to filter the final clustering results. Density scale based on reachability distance; For clusters Density threshold when generating in a hierarchical tree; For sample points De-cluster The density threshold at that time.

[0056] This stability score reflects the "lifetime" of a cluster within the hierarchical tree. Higher stability indicates stronger representativeness of the cluster. This invention selects the cluster with the highest stability as the final result and automatically removes noisy points, thereby ensuring the robustness and representativeness of the clustering results.

[0057] Step 2.3: Map to a subset of operating conditions;

[0058] After clustering, the data in Category I (PV operating period) and Category II (PV shutdown period) are divided into multiple operating condition subsets based on clustering labels, formally defined as follows:

[0059] (6);

[0060] (7);

[0061] in, The cluster label for the u-th sample is output by HDBSCAN; This is the p-th subset of operating conditions during the Class I time period (photovoltaic operating period); This is the m-th subset of operating conditions during the Class II time period (photovoltaic shutdown period); and These represent the number of operating condition clusters divided into Class I and Class II time periods, respectively; OpSt is the photovoltaic operating status flag (1 indicates photovoltaic operation, 0 indicates photovoltaic shutdown).

[0062] Ultimately, each subset of operating conditions or This corresponds to a subsequent missing value imputation sub-model. Through this two-level clustering process, the present invention achieves a layer-by-layer division from the overall photovoltaic operating status to fine-grained operating condition patterns, thereby providing a highly consistent data foundation for missing data imputation based on adaptive multi-model LSTM for different operating conditions.

[0063] 3. Multi-model filling framework for different working conditions.

[0064] After completing the primary photovoltaic operating status classification and the secondary HDBSCAN operating condition classification, this invention proposes a multi-model imputation framework for different operating conditions, addressing varying degrees of data gaps. This framework utilizes an adaptive structure switching mechanism to dynamically select appropriate model structures, thereby achieving accurate prediction and imputation of missing data.

[0065] 3.1 Adaptive structure switching mechanism based on missing length;

[0066] Let the length of the consecutive missing interval be L. m To avoid the subjectivity of manually setting thresholds, this invention introduces an adaptive threshold method based on data statistics.

[0067] First, collect the lengths of all missing intervals to form a set:

[0068] (8);

[0069] in, Indicates the first The length of a consecutive missing interval This represents the total number of missing intervals.

[0070] Calculate the quantile of the distribution for this set: 50th quantile Q 50 With the 90th percentile Q 90 Based on this, the segmentation threshold for missing length is defined as follows:

[0071] (9);

[0072] Wherein, L1 is the threshold used to distinguish between small spans and medium spans of missing data; L2 is the threshold used to distinguish between medium spans and large spans of missing data.

[0073] Ultimately, the model selection rule is defined as follows:

[0074] (10);

[0075] Among them, LSTM (Long Short-Term Memory Network) is a variant of Recurrent Neural Network (RNN). By introducing input gate, forget gate and output gate structure, it effectively solves the gradient vanishing and gradient explosion problems in long sequence training and can model the long-term dependencies of time series.

[0076] Attention-LSTM (Attention-based LSTM) introduces an attention mechanism on top of LSTM. By calculating the importance weights of the hidden states at each time step, it highlights the contribution of key moments and improves the accuracy of filling in missing data with a medium span.

[0077] BiLSTM-Attn (Bidirectional Attention-based LSTM) combines bidirectional LSTM with an attention mechanism to capture both sequential forward and backward dependencies and highlight key moment features, thereby improving the prediction accuracy and stability of long-span missing data.

[0078] This threshold setting method based on statistical quantiles can adapt to the actual missing data distribution of different stations or data sources, thereby ensuring the universality and robustness of model switching.

[0079] 3.2 LSTM model (suitable for small gaps);

[0080] For cases with a small number of consecutive missing points (L) m For data ≤L1), this invention employs a standard Long Short-Term Memory (LSTM) network model for data imputation. Its core idea is to utilize the gating mechanism of LSTM to capture short-term dependencies in local time series, thereby achieving high-precision prediction of missing data with small spans. This involves the following five steps:

[0081] Step 1: Define the input matrix:

[0082] The input matrix for type I time period is:

[0083] (11);

[0084] The input matrix for type II time period is:

[0085] (12);

[0086] Wherein, U, I, P and Q are the load voltage, current, active power and reactive power of phase A of the transformer, respectively; , , and These are the voltage, current, active power, and reactive power output by the inverter on the photovoltaic equipment side, respectively.

[0087] Step 2: Represent the time series:

[0088] For each working condition M i,j (i=1, 2, j=1, 2,...,K i If Mi,j Given N time series, each of length T, the nth time series can be represented as:

[0089] (13);

[0090] in, It is the input vector at time t.

[0091] Step 3: Train the corresponding LSTM model:

[0092] For each time step t, the LSTM state update formula is:

[0093] (14);

[0094] in, This represents the hidden state of the LSTM at time t. This represents the current input vector. This represents the hidden state of the previous time step. This indicates the cell state at the previous time step. This represents the nonlinear mapping function of LSTM.

[0095] For each time step t, the output of the LSTM for:

[0096] (15);

[0097] Among them, W o Let b be the weight matrix of the output layer. o This is the bias vector for the output layer. For operating condition M... i,j The filled results for all time series are as follows:

[0098] (16);

[0099] In summary, we obtained the corresponding LSTM models under different working conditions. The LSTM model can capture the short-term dependencies of local time series and has high accuracy and efficiency in filling small missing segments.

[0100] 3.3 Attention-LSTM model (suitable for medium-span missing data);

[0101] When the length of the missing interval is L1≤L m When the time interval is ≤L2, traditional LSTM struggles to capture long-term dependencies. This invention introduces an attention mechanism on top of LSTM, assigning weights to different time steps to enhance the model's ability to perceive information at key moments, thereby improving the accuracy of filling in missing segments with medium spans.

[0102] Attention-LSTM is based on standard LSTM, and its hidden state update process is completely consistent with equation (14), which will not be repeated here. It is important to emphasize that Attention-LSTM updates the hidden states of all time steps... It serves as input to the subsequent attention layer, used to calculate the importance weights at different time steps.

[0103] After obtaining all hidden states, an attention layer is introduced to compute the weighted context vector:

[0104] (17);

[0105] in, Let be the hidden state of the LSTM at time s; and These are the parameters for the attention layer. Assign an attention scoring vector; Let be the attention weight at time t; The weighted context vector serves as the final prediction input.

[0106] The final predicted value is obtained by combining the attention-weighted context vector. :

[0107] (18);

[0108] For operating condition M i,j After imputing all time series, the predicted result set is obtained. :

[0109] (19);

[0110] In summary, the Attention-LSTM model, by introducing an attention mechanism on top of LSTM, achieves focused attention on key time segments, and can significantly improve prediction accuracy and robustness in scenarios with medium-span missing data.

[0111] 3.4. BiLSTM-Attn model (suitable for large-span missing data);

[0112] When the length of the missing interval exceeds L2 (L m >L2), where dependencies may exist over longer time spans. In such cases, unidirectional LSTM or Attention-LSTM struggle to fully utilize bidirectional information. This invention employs a bidirectional long short-term memory network (BiLSTM) combined with an attention mechanism (Attn). By simultaneously considering forward and backward temporal features and introducing attention weighting, it improves the prediction accuracy for data with large spans of missing information.

[0113] Unlike unidirectional LSTM, BiLSTM includes forward LSTM and backward LSTM:

[0114] (20);

[0115] in, This represents the hidden state of the forward LSTM at time t. This represents the hidden state of the backward LSTM at time t. and These represent the forward and backward LSTM networks, respectively.

[0116] The final hidden state For forward and backward splicing:

[0117] (twenty one);

[0118] Meanwhile, based on the introduction of the attention mechanism, we substitute equation (21) into equation (17), and then substitute the result into equations (18) and (19) to obtain the final filling result.

[0119] Through the above steps, this invention constructs a complete framework for filling in missing data under different work conditions:

[0120] (1) First-level classification: The data is divided into Class I (photovoltaic working period) and Class II (photovoltaic shutdown period) according to the photovoltaic operating status (OpSt) to ensure the overall consistency of the dataset under different operating modes.

[0121] This invention is based on a first-level classification mechanism of photovoltaic operating status. By using the photovoltaic operating status (OpSt) recorded by SCU, the data is divided into photovoltaic working period (Class I) and photovoltaic shutdown period (Class II), which ensures the overall consistency of the dataset and lays the foundation for subsequent modeling.

[0122] (2) Secondary partitioning: The HDBSCAN clustering algorithm is used to refine the working conditions within each data category, automatically identify different working condition modes, and remove noise points to ensure that subsequent modeling is carried out on homogeneous data.

[0123] This invention is based on the HDBSCAN two-level working condition classification method. It introduces the HDBSCAN algorithm to perform fine-grained clustering within various types of data, automatically identifies working condition patterns and removes noise points, avoids the subjectivity caused by manually setting the number of clusters K, and improves the stability and representativeness of the clustering results.

[0124] (3) Multi-model framework:

[0125] For different missing spans, an adaptive structure switching mechanism based on statistical quantiles is proposed;

[0126] 1) For small-span missing data, the LSTM model is used;

[0127] 2) For missing data of moderate span, the Attention-LSTM model is used;

[0128] 3) For large-span missing data, the BiLSTM-Attn model is used;

[0129] The three types of models are dynamically scheduled under a unified framework to adapt to different missing scenarios.

[0130] This invention employs an adaptive structure switching mechanism based on missing length: through statistical distribution (median Q) 50 High quantile Q 90 Determine thresholds L1 and L2, based on the consecutive missing length L. m Dynamically selecting the model structure enables adaptive switching to fill in missing values. The introduction of the Attention-LSTM mechanism adds an attention mechanism to the LSTM, assigning higher weights to the hidden states at key time steps, improving the model's prediction accuracy and robustness under moderate missing value. The combination of BiLSTM and the attention mechanism utilizes a bidirectional LSTM to simultaneously model forward and backward dependencies, and introduces an attention mechanism to enhance the ability to capture large missing values, ensuring the accuracy and stability of the prediction results.

[0131] like Figure 2 As shown, on the same set of transformer area current data, compared with traditional missing data imputation methods such as MICE, RF-Impute, Transformer, and single-structure LSTM, Attention-LSTM, and BiLSTM-Attn, the power monitoring data missing data imputation method proposed in this invention, which combines HDBSCAN and a multi-model mechanism, exhibits higher fitting accuracy and temporal consistency in missing data scenarios with small, medium, and large spans. This is achieved by using the median Q of the missing length... 50 and high quantile Q 90 Adaptively determining thresholds L1 and L2 avoids the severe lag and distortion of traditional single models under large-span missing data, and also overcomes the problems of overfitting and oscillation of complex models in small-span missing data scenarios. Experimental results show that the filling curves of the method in the current large fluctuation range and the nighttime small fluctuation range highly coincide with the real data, significantly reducing peak bias and local noise. Compared with the comparison method, the overall error is smaller and the curve is smoother and more continuous, balancing prediction accuracy and robustness.

Claims

1. A method for filling missing power monitoring data by combining HDBSCAN and a multi-model mechanism, characterized in that, Includes the following steps: Step 1: Divide the photovoltaic operating status data of the transformer substation into primary categories according to the photovoltaic operating status recorded by the intelligent integrated terminal: If the photovoltaic equipment is in operation, it enters the Class I time period, which includes both photovoltaic data and transformer data; if the photovoltaic equipment is not in operation, it enters the Class II time period, which includes only transformer data. Step 2: Within each data category, the HDBSCAN algorithm is used to perform secondary partitioning of the data. Based on the local density differences in the sample distribution, cluster structures are automatically formed, and noise points are automatically identified to achieve a refined division of operating conditions. Step 3: Construct an adaptive structure switching mechanism based on missing data length. According to the length of the missing interval of the transformer area data and the type of working condition, dynamically select LSTM, Attention-LSTM or BiLSTM-Attn model for training, and model the characteristics of different working conditions and missing data spans in a targeted manner to achieve prediction and filling of missing data. Step 3 involves constructing an adaptive structure switching mechanism based on missing length, specifically as follows: Let the length of the consecutive missing interval be L. m We introduce an adaptive thresholding method based on data statistics; first, we collect the lengths of all missing intervals to form a set: （7）； in, Indicates the first The length of a consecutive missing interval This represents the total number of missing intervals. The distribution quantiles of the set are calculated, and the segmentation thresholds for the missing length are defined according to the number of quantiles: L1 is the threshold used to distinguish between small-span and medium-span missing lengths; L2 is the threshold used to distinguish between medium-span and large-span missing lengths. The model selection rule is defined as follows: （8）； Among them, LSTM is the standard long short-term memory network, Attention-LSTM is a long short-term memory network that introduces an attention mechanism, and BiLSTM-Attn is a bidirectional attention long short-term memory network.

2. The power monitoring data missing completion method combining HDBSCAN and multi-model mechanism according to claim 1, characterized in that, In step 2, the HDBSCAN algorithm is used to perform two-level partitioning of the power data, specifically as follows: Step 2.1: Construct the density graph and minimum spanning tree; Determine the key parameters of the HDBSCAN algorithm, including the minimum number of samples min_samples and the minimum cluster size min_cluster_size; and calculate the core distance core_dist and the mutual reach distance mreach_dist for each sample point. （1）； （2）； in, Let u be the feature vector of the u-th sample; and Representing sample points respectively and The Euclidean distance to its k-th nearest neighbor; Represents sample points and The Euclidean distance between them; Then, a weighted graph is constructed based on the mutual reachability distance, and Kruskal's algorithm is used to generate the minimum spanning tree; the optimization objective MST of the minimum spanning tree is: （3）； Where G represents a weighted graph consisting of sample points; Tr represents a candidate spanning tree, which is a set of edges in the weighted graph G that can connect all sample points without forming loops. The weights of the edges, i.e., the sample points. and The reachability distance between each other; Step 2.2: Construct a density hierarchy tree and extract stable clusters; Based on the generated minimum spanning tree, a density hierarchy tree is constructed by progressively increasing the distance threshold, and effective clusters are automatically selected using a cluster stability score; the cluster stability score is defined as: （4）； in, The stability score for cluster C is used to filter the final clustering results; Density scale based on reachability distance; The density threshold for cluster C when it is generated in the hierarchical tree; For sample points The density threshold when detached from cluster C; Step 2.3: Map to a subset of operating conditions; After clustering, the data in time periods I and II are divided into multiple working condition subsets based on the cluster labels, as formally defined below: （5）； （6）； in, Let be the cluster label of the u-th sample; This is the p-th subset of working conditions in time period I; This is the m-th subset of working conditions in time period II; and These represent the number of operating condition clusters divided into Class I and Class II time periods, respectively; OpSt is the photovoltaic operating status indicator. Each operating condition subset or This will correspond to a subsequent missing value imputation sub-model.

3. The power monitoring data missing completion method combining HDBSCAN and multi-model mechanism according to claim 2, characterized in that, Imputation using a standard Long Short-Term Memory (LSTM) network model leverages gating mechanisms to capture short-term dependencies in local time series, enabling high-precision prediction of missing data with small spans. Specifically, this includes: Step 3.1.1: Define the input matrix; The input matrix for type I time period is: （9）； The input matrix for type II time period is: （10）； Wherein, U, I, P and Q are the load voltage, current, active power and reactive power of phase A of the transformer, respectively; , , and These are the voltage, current, active power, and reactive power output by the inverter on the photovoltaic equipment side, respectively. Step 3.1.2: Represent the time series; For each working condition M i,j If M i,j Given N time series, each with a length of T, find the nth time series. Represented as: （11）； in, This is the input vector at time t; i = 1, 2, representing the two types of time intervals; j = 1, 2, ..., K i , representing the ordinal number of the subset of functions; Step 3.1.3: Train the corresponding LSTM model; For each time step, the LSTM state update formula is: （12）； in, This represents the hidden state of the LSTM at time t. This represents the current input vector. This represents the hidden state of the previous time step. This indicates the cell state at the previous time step; Represents the nonlinear mapping function of LSTM; For each time step, the LSTM output for: （13）； Among them, W o Let b be the weight matrix of the output layer. o This is the bias vector for the output layer; For operating condition M i,j Filled results for all time series for: （14）； in, This is the filling result for the nth time series obtained from the LSTM model.

4. The power monitoring data missing completion method combining HDBSCAN and multi-model mechanism according to claim 3, characterized in that, When using a long short-term memory network with an attention mechanism for imputation, the hidden states h of all time steps are... t As input to the subsequent attention layer, it is used to calculate the importance weights at different time steps; After obtaining all hidden states, an attention layer is introduced to compute the weighted context vector: （15）； in, and Here are the parameters for the attention layer; v is the attention scoring vector. Let be the attention weight at time t; The weighted context vector serves as the final prediction input; Let be the hidden state of the LSTM at time s; The final predicted value is obtained by combining the attention-weighted context vector. : （16）； For operating condition M i,j After imputing all time series, the predicted result set is obtained. : （17）； in, This is the filling result for the nth time series obtained by the Attention-LSTM model.

5. The power monitoring data missing completion method combining HDBSCAN and multi-model mechanism according to claim 4, characterized in that, When using a bidirectional attentional long short-term memory network for imputation, the temporal features of both forward and backward directions are considered simultaneously, and attentional weighting is introduced. Bidirectional Long Short-Term Memory (LSTM) networks consist of forward LSTM and backward LSTM: （18）； in, For the forward LSTM at time The hidden state; For backward LSTM at time The hidden state; and These represent the forward and backward LSTM networks, respectively. The final hidden state For forward and backward splicing: （19）； Meanwhile, based on the introduction of the attention mechanism, we substitute equation (19) into equation (15), and then substitute the result into equations (16) and (17) to obtain the final filling result.

Images