Intelligent aggregation method and system for multi-source heterogeneous data based on hierarchical time sequence multi-scale learning

Abstract

The present application relates to the field of power information system and artificial intelligence technology, and more particularly to a layered time series multi-scale learning multi-source heterogeneous data intelligent aggregation method and system. The present application extracts structured, time series, unstructured data features and fuses to generate comprehensive representation and layered label portrait; extract time series multi-scale features and distribution network topology spatial features, and fuse to obtain space-time joint representation; based on Markov decision process and Dueling Double DQN, train the aggregation decision network to output the optimal action; perform three-layer progressive quality check and graded processing of abnormal data; through incremental online learning, drive the aggregation strategy and the check rule to continue iteration. The purpose of the present application is to provide a layered time series multi-scale learning multi-source heterogeneous data intelligent aggregation method and system, to solve the problems of difficult efficient automatic aggregation of multi-source heterogeneous data, insufficient cross-scale feature fusion capability and lack of adaptive dynamic adjustment capability of aggregation strategy in power supply service scenarios.

Description

Technical Field

[0001] This invention relates to the fields of power information systems and artificial intelligence technology, and in particular to a method and system for intelligent aggregation of multi-source heterogeneous data using hierarchical temporal multi-scale learning. Background Technology

[0002] The modern power supply service system encompasses distribution network operation monitoring, fault maintenance and repair, asset and equipment management, and customer service. The system encompasses various business functions such as service management and energy efficiency management. Over its long-term independent evolution, each business system has generated a large amount of multi-source, heterogeneous data. This data exhibits significant differences in timeliness, format, collection frequency, and semantic granularity, specifically in the following three aspects: 1. The power supply service data sources cover dispatch automation system (SCADA), distribution transformer terminal (TTU), electricity consumption information collection system, production management system (PMS), customer service system (CRM) and external meteorological and geographic information platforms, etc. Each system adopts different data protocols (IEC61968 / 61970, MQTT, Modbus, HTTPREST, etc.) and data formats (relational tables, time-series streams, JSON / XML semi-structured, image / text unstructured), resulting in a high degree of heterogeneity.

[0003] 2. The core monitoring data collection frequency can reach the second level or even the sub-second level, the load forecasting data is granular at 15 minutes or hours, and the asset management data is updated at the daily level or periodically. The fusion and calculation of data at different time scales faces technical challenges such as time alignment and scale normalization.

[0004] 3. Due to factors such as equipment aging, network interruption, transmission delay, and acquisition error, the actual acquired data generally suffers from quality problems such as missing, drifting, anomalies, and conflicts, which seriously restricts the reliability of data-driven analysis.

[0005] Traditional ETL (Extract-Transform-Load) pipelines use static rules and batch scheduling modes, which are difficult to adapt to the dynamic changes and real-time requirements of power supply service data. At the same time, existing data lake / data warehouse solutions lack specialized processing capabilities for time-series multi-scale features, and cross-scale feature extraction relies on a large amount of manual feature engineering.

[0006] Therefore, designing a hierarchical temporal multi-scale learning method and system for intelligent aggregation of multi-source heterogeneous data that can solve the above-mentioned technical problems is a technical issue that needs to be addressed. Summary of the Invention

[0007] To address the aforementioned issues, the present invention aims to provide a hierarchical, time-series, multi-scale learning-based intelligent aggregation method and system for multi-source heterogeneous data, solving the problems of inefficient and automatic aggregation of multi-source heterogeneous data in power supply service scenarios, insufficient cross-scale feature fusion capabilities, and a lack of adaptive dynamic adjustment capabilities in aggregation strategies.

[0008] In a first aspect, the present invention provides a hierarchical temporal multi-scale learning-based intelligent aggregation method for multi-source heterogeneous data, characterized by comprising the following steps: Step S1: Extract features from structured data, time-series data, and unstructured data respectively, and fuse them through a cross-modal attention mechanism to obtain a comprehensive representation of the data source; Step S2: Based on the comprehensive characterization, a hierarchical tag generation framework is used to automatically generate basic data tags, quality assessment tags, and business scenario tags for the data source, thus forming a data source tag profile. Step S3: Perform adaptive pooling operations on the temporal features at M preset time scales to generate multi-scale representations. Obtain temporal multi-scale features by weighted fusion with learnable scale attention weights. Extract the distribution network topology spatial features using a graph convolutional network. Then, fuse the temporal multi-scale features with the spatial topology features through a bilinear attention mechanism to obtain a spatiotemporal joint representation. Step S4: Construct a mathematical model of the Markov decision process, build a state vector with spatiotemporal joint representation, data source label profile and system resource state, design a multi-dimensional composite reward function, and use dual-network deep Q-learning combined with priority experience replay to train the aggregation decision network and output the optimal aggregation action. Step S5: Perform field-level numerical verification, record-level business rule verification, and dataset-level global statistical verification in sequence, and process the detected abnormal data according to the risk level. Step S6: Combining the verification results and quality assessment report, the aggregation strategy is optimized and the verification rules are iterated through incremental online learning.

[0009] Furthermore, in step S1, features are extracted from structured data, time-series data, and unstructured data, specifically including: For the structured data branch, a multilayer perceptron with residual connections is used to extract numerical features, and a learnable embedding mapping is performed on the discrete category features. The numerical features and the embedded features are then concatenated to obtain the structured features. For the temporal data branch, a bidirectional long short-term memory network is used to capture long-term temporal dependencies. A multi-head temporal attention mechanism is introduced to extract key moment features. Three different one-dimensional convolutional layers with different kernel sizes are designed in parallel to extract multi-scale local temporal patterns. Attention features and convolutional features are concatenated to obtain temporal features. For unstructured data classification, a pre-trained BERT model is used to extract text semantic features, and ResNet-50 is used to extract image visual features. The text features and image features are then concatenated to obtain unstructured features.

[0010] Furthermore, in step S2, generating a frame through hierarchical tags specifically includes: For the basic attribute label layer, a softmax classifier based on structured features outputs labels for data type, scale level, and update frequency. For the quality assessment label layer, the regression header of the time series features outputs data integrity, reliability, and consistency scores; For the business scenario label layer, based on unstructured features, a graph neural network is used to construct semantic constraint relationships between labels and output the business scenario label probability. A three-layer label generation network is jointly trained using a multi-task learning loss function, where the loss function is:

[0011] in, , , These are cross-entropy, mean squared error, and binary cross-entropy loss, respectively. This is a hyperparameter for task weights. It is the set of all learnable parameters.

[0012] Furthermore, in step S3, adaptive pooling operations are performed on the temporal features at M preset time scales to generate multi-scale representations. The temporal multi-scale features are then obtained through weighted fusion using learnable scale attention weights. Specifically, this includes: First, a set of time scales covering all scenarios of power business is predefined. S1 to SM correspond to time granularities of seconds, hours, days, and months, respectively, and M is the total number of time scales. Secondly, for the input raw time series data At each scale The pooling window size is automatically adjusted through adaptive pooling operations to generate a single-scale feature representation that matches the time scale, as follows:

[0013] Secondly, for each single-scale feature representation Through the corresponding m-th scale feature extraction network Deep feature extraction is performed to obtain single-scale deep features. ; Secondly, all single-scale deep features are input into a learnable scale attention layer, and the attention weights corresponding to each time scale are calculated through linear transformation and softmax activation function. And all weights satisfy the normalization constraint. ; Finally, the weighted fusion of each single-scale depth feature with its corresponding attention weight is multiplied and summed to complete the weighted fusion of cross-scale features, resulting in the final temporal multi-scale fusion feature formula as follows:

[0014] Among them, weight It is obtained through a learnable linear transformation plus softmax. is the feature extraction network at scale m.

[0015] Furthermore, in step S4, the data aggregation problem is formalized as a Markov decision process quintuple, including S, A, P, R, and Y, with the specific definitions of each component as follows: The state space S is characterized by spatiotemporal joint features. Label vectors Quality score vector Concatenate with system resource state vector constitute; Action space A includes aggregation operations that merge two data sources into the same aggregation group, separation operations that separate data sources from the current aggregation group, and refactoring operations that reorganize the global aggregation scheme according to changes in business scenarios. The reward function R is designed as a multi-dimensional composite function to comprehensively evaluate data quality, aggregation efficiency, semantic consistency, and resource consumption, guiding the agent to learn the optimal aggregation strategy. The specific formula is as follows:

[0016] in, Greater than or equal to 0 and It equals 1, and is dynamically adjusted based on business priorities.

[0017] Furthermore, in step S4, a dual-network deep Q-learning combined with priority experience replay is used to train the aggregation decision network, specifically including: First, the spatiotemporal joint representation matrix output in step S3 and the adjacency matrix of the current aggregation state are input into the graph-integrated convolutional network to extract the structured state code that simultaneously contains the spatiotemporal features of the data and the aggregation structure information. Secondly, the structured state code is input into the Dueling backbone network and decomposed into state values. and dominance function The overall value of the current state and the relative advantage of each aggregate action are output separately, and the final Q value is obtained through combination calculation. The specific formula is as follows:

[0018] Secondly, set up online networks with completely identical structures. and target network The online network is responsible for selecting the optimal aggregation action in real time, while the target network is responsible for calculating a stable target Q value, thus separating the action selection and target Q value calculation processes. Finally, an experience replay pool is constructed to store transfer samples of agent-environment interaction. Different sampling priorities are assigned to samples according to the magnitude of TD error, important samples are given higher sampling weights, and sampling bias is corrected through importance sampling.

[0019] Furthermore, step S5 specifically includes: The field-level numerical verification employs an improved interquartile range (ICM) method for adaptive detection of numerical anomalies. The detection coefficient k is dynamically adjusted based on the kurtosis of the data distribution, as shown in the following formula: Combined with isolated forest detection of high-dimensional feature anomalies; The record-level business rule verification checks the cross-field logical relationship and temporal continuity of data records, and uses a multi-classifier integration to identify error types; The dataset-level global statistical verification is based on a statistical process control framework. It uses an LSTM time-series anomaly detection model to monitor changes in key statistics of the dataset and identifies anomalous data clusters through DBSCAN clustering.

[0020] Furthermore, in step S6, the incremental online learning-driven optimization of the aggregation strategy and the iteration of the validation rules specifically include: Incremental state updates employ a sliding window mechanism, with different window sizes set for different business scenarios. Feature updates within the window utilize an exponentially weighted moving average, as shown in the following formula:

[0021] in, The forgetting factor is adaptively set according to the business scenario; the strategy update adopts a gradient accumulation method, and the parameter update is performed once every K steps. The base learning rate is dynamically adjusted according to the business effect evaluation.

[0022] Secondly, the present invention provides a multi-source heterogeneous data intelligent aggregation system for hierarchical temporal multi-scale learning, including a multimodal data active perception and adaptive access module, a multi-scale feature extraction and representation learning module, a data aggregation decision optimization module, and a multi-level joint verification and feedback module. The multimodal data active perception and adaptive access module is used to collect multi-source heterogeneous power data, extract modal features of structured, time-series and unstructured data respectively, and generate a comprehensive representation of the data source through cross-modal attention mechanism; The multi-scale feature extraction and representation learning module is used to adaptively pool time-series data under multiple preset time scales, generate time-series multi-scale features through learnable scale attention fusion, extract distribution network topology spatial features using graph convolutional networks, and generate spatiotemporal joint representations through bilinear attention fusion of spatiotemporal features. The data aggregation decision optimization module is used to model the data aggregation problem as a Markov decision process, construct a system state vector containing spatiotemporal features, labels, and resource states; design a multi-dimensional composite reward function; use DuelingDoubleDQN combined with priority experience replay to train the decision agent; and dynamically output the optimal aggregation action through incremental online learning. The multi-level joint verification and feedback module is used to perform three-level progressive quality verification at the field level, record level, and dataset level; and to process abnormal data according to risk level.

[0023] The present invention has the following beneficial effects: 1-This invention uses a three-branch parallel feature recognition network to uniformly process three types of data modalities: structured, temporal, and unstructured. It achieves effective integration of heterogeneous information through cross-modal attention fusion, thereby improving the recognition accuracy of multi-source data.

[0024] 2- This invention employs a hierarchical temporal multi-scale representation learning framework that can simultaneously capture multi-granularity temporal features ranging from seconds to months, overcoming the limitations of traditional methods that model at fixed time scales and reducing the MAE of typical time series analysis tasks such as load forecasting.

[0025] 3- This invention uses the DuelingDoubleDQN aggregation decision model to achieve end-to-end learning of aggregation strategies within the MDP framework. By combining priority experience replay and incremental online learning, the aggregation scheme can adaptively respond to dynamic changes in the data source, thereby improving the overall data quality score of the system compared to static rule aggregation. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of Embodiment 2 of the present invention. Detailed Implementation

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Example 1 See Figure 1As shown, it includes the following steps: Step S1: Extract features from structured data, time-series data, and unstructured data respectively, and fuse them through a cross-modal attention mechanism to obtain a comprehensive representation of the data source; Step S2: Based on the comprehensive characterization, a hierarchical tag generation framework is used to automatically generate basic data tags, quality assessment tags, and business scenario tags for the data source, thus forming a data source tag profile. Step S3: Adaptive pooling is performed on the temporal features at M preset time scales to generate multi-scale representations. These multi-scale temporal features are then obtained through weighted fusion using learnable scale attention weights. A graph convolutional network (GCN) is used to extract the distribution network topology spatial features, and a bilinear attention mechanism is used to fuse the temporal multi-scale features with the spatial topology features to obtain a spatiotemporal joint representation. Specifically, a graph convolutional network (GCN) is used to extract the distribution network topology features. Let the distribution network diagram... Where V is the set of device nodes. It is an adjacency matrix. The propagation rule for layer l is:

[0028] in, To add a self-loop to the adjacency matrix, This is the corresponding degree matrix. This represents the learnable weight matrix for the l-th layer. Node feature matrix. It is composed of raw features such as sensor measurements and device attributes. After passing through an L-layer GCN, a spatial feature representation with fused topological structure information is obtained. .

[0029] Multi-scale temporal features Spatial topological features Spatiotemporal fusion using a bilinear attention mechanism:

[0030] Among them, multi-head attention , For the Key dimension.

[0031] Step S4: Construct a mathematical model of the Markov decision process, build a state vector with spatiotemporal joint representation, data source label profile and system resource state, design a multi-dimensional composite reward function, and use dual-network deep Q-learning combined with priority experience replay to train the aggregation decision network and output the optimal aggregation action. Step S5: Perform field-level numerical verification, record-level business rule verification, and dataset-level global statistical verification sequentially. Process detected abnormal data according to its risk level; specifically, define abnormal events. ,in , .

[0032] For low-risk anomalies The system automatically triggers a repair strategy. For missing values, a data repair model based on Generative Adversarial Networks (GAIN) is used:

[0033] The generator G takes masked data and random noise as input, and the discriminator distinguishes between observed values ​​and filled values ​​through a hint mechanism. This is to indicate the weight hyperparameter.

[0034] For high-risk anomalies The system generates a detailed anomaly report and notifies operations and maintenance personnel through a multi-level alarm mechanism (SMS / email / system message). After manual confirmation, the system performs modification or removal operations to ensure that core business data is not damaged.

[0035] Rule optimization is driven by a deep Q-network (DQN). The current rule execution effect (accuracy) is then considered. Recall rate F1 score The DQN is constructed with state vectors as input. The action space includes operations such as parameter threshold adjustment, weight coefficient update, rule merging and splitting. The reward function comprehensively considers rule effect improvement and resource consumption.

[0036] in, This represents the efficiency trade-off coefficient. The experience replay pool stores historical optimization experiences, and the training objective is to minimize the Bellman error, thereby achieving intelligent and continuous evolution of the rule system.

[0037] Step S6: Combining the verification results and quality assessment report, the aggregation strategy is optimized and the verification rules are iterated through incremental online learning.

[0038] Furthermore, in step S1, features are extracted from structured data, time-series data, and unstructured data, specifically including: For the structured data branch, a multilayer perceptron with residual connections is used to extract numerical features. A learnable embedding mapping is applied to the discrete category features, and the numerical features are concatenated with the embedded features to obtain structured features. Specifically, this is applied to structured inputs such as equipment ledgers and measurement data. A multilayer perceptron (MLP) with residual connections is used, and the specific formula is as follows:

[0039] in, Let l be the weight matrix of the l-th layer. For discrete category features (such as device type and voltage level), design learnable embedding maps:

[0040] in, Let be the value of the k-th category feature. For the number of categories, For the embedding dimension. The final structured feature is the concatenation of each numerical feature and the embedding feature: .

[0041] For the temporal data branch, a bidirectional long short-term memory network is used to capture long-term temporal dependencies. A multi-head temporal attention mechanism is introduced to extract key moment features. Three different one-dimensional convolutional layers with different kernel sizes are designed in parallel to extract multi-scale local temporal patterns. Attention features and convolutional features are concatenated to obtain temporal features. Specifically, for time-series inputs such as load curves and equipment operation records... Bidirectional Long Short-Term Memory (BiLSTM) network is used to capture long-range dependencies:

[0042] Building upon this, a multi-head temporal attention mechanism is introduced to extract key moment features:

[0043] To capture multi-scale local temporal patterns, three convolutional kernel sizes were designed in parallel. The one-dimensional convolutional layer extracts temporal features of different granularities and then aggregates them through max pooling:

[0044] Final characteristics of temporal branching: .

[0045] For unstructured data classification, a pre-trained BERT model is used to extract semantic features from the text, and ResNet-50 is used to extract visual features from the image. The text features and image features are then concatenated to obtain the unstructured features. Specifically, for texts such as fault reports and operation logs, a pre-trained BERT model is used to extract semantic features, and the hidden state of the [CLS] token is taken as the sentence-level representation.

[0046] Visual features were extracted from the device images using ResNet-50 and then obtained after global average pooling:

[0047] The three-branch features are used to obtain a comprehensive representation of the data source through cross-modal attention fusion. This serves as input for subsequent label generation and aggregation decisions.

[0048] Furthermore, in step S2, generating a frame through hierarchical tags specifically includes: For the basic attribute label layer, a softmax classifier using structured features outputs labels for data type, scale level, and update frequency; specifically, the basic attribute label layer... Includes data types , Through structured branching features The output of the softmax classifier:

[0049] For the quality assessment label layer, data integrity, reliability, and consistency scores are output through the regression head of time-series features; specifically, the quality assessment label layer... Utilizing temporal branching features Assessment completeness Credibility and consistency The regression head outputs a score in the [0, 1] interval:

[0050] in, This is the Sigmoid function.

[0051] For the business scenario tagging layer, based on unstructured features, a graph neural network is used to construct semantic constraint relationships between tags and output the probability of business scenario tags; specifically, the business scenario tagging layer... Based on unstructured branch features We model the semantic constraints between labels using graph neural networks (GNNs). Let the label dependency graph be... ,in For label nodes, To represent label co-occurrence or implication relationships, the GNN message passing process is as follows:

[0052] in, Let i be the set of neighboring nodes of node i. Finally, the probability of each scene label is output by classifying the nodes.

[0053] A three-layer label generation network is jointly trained using a multi-task learning loss function, where the loss function is:

[0054] in, , , These are cross-entropy, mean squared error, and binary cross-entropy loss, respectively. This is a hyperparameter for task weights. It is the set of all learnable parameters.

[0055] Specifically, to reduce the dependence on annotations, a three-stage self-supervised learning framework of pre-training, fine-tuning, and online optimization is designed.

[0056] During the pre-training phase, corresponding self-supervised tasks are designed for each branch: the structured branch is reconstructed using random feature masks, with a masking rate of... Minimize reconstruction error:

[0057] The time-series branch employs contrastive learning, treating different time windows from the same data source as positive sample pairs and randomly sampling other data sources as negative samples. The contrastive loss is:

[0058] in, For cosine similarity, Let N be the temperature coefficient and N be the batch size. Unstructured branches employ cross-modal alignment to minimize the text-image feature distance from the same data source.

[0059] The fine-tuning phase employs a layered unfreezing strategy, gradually unfreezing network parameters from the top to the bottom, combined with supervised loss based on labeled data. Weighted combination with self-supervised loss:

[0060] in The model decays with cosine annealing during training epochs, allowing for a smooth transition to supervised learning. In the online optimization phase, unlabeled samples with the highest information content are selected through active learning, with the selection criterion based on Bayesian uncertainty estimation.

[0061] in, Let be the information entropy, and D be the labeled dataset. Selected samples are added to the training set after being manually confirmed, driving the continuous evolution of the model.

[0062] Furthermore, in step S3, adaptive pooling operations are performed on the temporal features at M preset time scales to generate multi-scale representations. The temporal multi-scale features are then obtained through weighted fusion using learnable scale attention weights. Specifically, this includes: First, a set of time scales covering all scenarios of power business is predefined. S1 to SM correspond to time granularities of seconds, hours, days, and months, respectively, and M is the total number of time scales. Secondly, for the input raw time series data At each scale The pooling window size is automatically adjusted through adaptive pooling operations to generate a single-scale feature representation that matches the time scale, as follows:

[0063] Secondly, for each single-scale feature representation Through the corresponding m-th scale feature extraction network Deep feature extraction is performed to obtain single-scale deep features. ; Secondly, all single-scale deep features are input into a learnable scale attention layer, and the attention weights corresponding to each time scale are calculated through linear transformation and softmax activation function. And all weights satisfy the normalization constraint. ; Finally, the weighted fusion of each single-scale depth feature with its corresponding attention weight is multiplied and summed to complete the weighted fusion of cross-scale features, resulting in the final temporal multi-scale fusion feature formula as follows:

[0064] Among them, weight It is obtained through a learnable linear transformation plus softmax. is the feature extraction network at scale m.

[0065] Furthermore, in step S4, the data aggregation problem is formalized as a Markov decision process quintuple, including S, A, P, R, and Y, with the specific definitions of each component as follows: The state space S is characterized by spatiotemporal joint features. Label vectors Quality score vector Concatenate with system resource state vector Composition; specifically, system state It consists of the combined representation vectors of all data sources at the current moment:

[0066] in, It is a spatiotemporal joint feature. This is the current label vector. This is a quality scoring vector (completeness / reliability / consistency). This is a system resource state vector (CPU utilization, memory usage, network bandwidth, etc.).

[0067] Action space A includes aggregation operations that merge two data sources into the same aggregation group, separation operations that separate data sources from the current aggregation group, and refactoring operations that reorganize the global aggregation scheme according to changes in business scenarios; specifically, aggregation actions... Defined as a collection of data sources Grouping operations. Using discrete actions as representations, aggregation actions are divided into three categories: (1) Aggregation operation : Data source and Merge into the same aggregation group; (2) Separation operation : Data source Separate from the current aggregation group; (3) Reconstruction operation Reorganize the global aggregation solution based on changes in business scenarios.

[0068] The reward function R is designed as a multi-dimensional composite function to comprehensively evaluate data quality, aggregation efficiency, semantic consistency, and resource consumption, guiding the agent to learn the optimal aggregation strategy. The specific formula is as follows:

[0069] in, Greater than or equal to 0 and Equal to 1, dynamically adjusted based on business priority. Specifically, the definitions of each component are as follows: Data Quality Bonus A comprehensive score measures the quality of the aggregated dataset:

[0070] in, For the group set in the current aggregation scheme, These represent the mean values ​​of completeness, reliability, and consistency for group g, respectively. .

[0071] Aggregate efficiency reward Measure the balance between latency and throughput in data processing:

[0072] Semantic consistency reward Tag similarity based on data sources within the group:

[0073] Resource consumption penalty Penalties will be imposed for exceeding resource thresholds:

[0074] hyperparameters Greater than or equal to 0 and It equals 1 and is dynamically adjusted based on business priorities.

[0075] Furthermore, in step S4, a dual-network deep Q-learning combined with priority experience replay is used to train the aggregation decision network, specifically including: First, the spatiotemporal joint representation matrix output from step S3 and the adjacency matrix of the current aggregation state are input into the graph-multiplexed convolutional network to extract the structured state code that simultaneously contains the spatiotemporal features of the data and the aggregation structure information; the specific formula is as follows:

[0076] in This is the adjacency matrix of the current aggregation state (data sources within a group are connected, while those between groups are disconnected).

[0077] Secondly, the structured state code is input into the Dueling backbone network and decomposed into state values. and dominance function The overall value of the current state and the relative advantage of each aggregate action are output separately, and the final Q value is obtained through combination calculation. The specific formula is as follows:

[0078] Secondly, set up online networks with completely identical structures. and target network The online network is responsible for selecting the optimal aggregation action in real time, while the target network is responsible for calculating a stable target Q value, thus separating the action selection and target Q value calculation processes. Finally, an experience replay pool is constructed to store transfer samples of agent-environment interactions. Different sampling priorities are assigned to samples based on the magnitude of the TD error, with higher sampling weights given to important samples. Sampling bias is corrected through importance sampling. The details are as follows:

[0079] Among them, priority , Let p be the priority hyperparameter, and TD error be the parameter. The parameter update objective is to minimize the weighted mean square TD error.

[0080] Importance sampling weight N is the playback pool size, and k is the correction coefficient.

[0081] Furthermore, step S5 specifically includes: The field-level numerical validation employs an improved interquartile range (ICM) method for adaptive detection of numerical anomalies. The specific formula is as follows: The detection coefficient k is dynamically adjusted based on the kurtosis of the data distribution, and the specific formula is as follows: ,in For sample kurtosis, and The scene-dependent hyperparameters allow the detection threshold to adaptively change with the data distribution. This is combined with the Isolation Forest algorithm to detect high-dimensional feature outliers; the isolation score is defined as:

[0082] in, Let x be the path length of sample x in the isolated tree. H (for harmonic series) E[h(x)] For the integrated mean. show x This is an outlier.

[0083] The record-level business rule verification checks the cross-field logical relationships and temporal continuity of data records, and uses a multi-classifier integration to identify error types; taking power balance constraints in a distribution network as an example, it verifies the monitoring records. r=(V,I,P,Q) Verification:

[0084] Among them, tolerance threshold Dynamically set according to equipment type and range. Verify the timing continuity of the timing record set:

[0085] in This is the maximum permissible rate of change (determined based on physical constraints). The sampling interval is defined as follows. Records that violate the above constraints are marked as logical anomalies, triggering error type identification in the multi-classifier ensemble:

[0086] The K base classifiers (random forest, gradient boosting tree GBM, and lightweight gradient boosting machine LightGBM) output the error type (missing / noise / conflict) through weighted voting, and the weights are dynamically updated based on the performance of the validation set.

[0087] The dataset-level global statistical verification, based on a statistical process control (SPC) framework, utilizes an LSTM time-series anomaly detection model to monitor changes in key statistics of the dataset and identifies anomalous data clusters through DBSCAN clustering. Specifically, at the dataset level, overall data quality is monitored based on the SPC framework. Let the dataset... The key statistic at time t is (Mean, variance, skewness, kurtosis), modeling statistical sequences using an LSTM-based time-series anomaly detection model:

[0088] When the actual statistic Deviation from predicted value exceeds When the confidence interval is reached, a global anomaly alarm is triggered:

[0089] After an alarm is triggered, the system automatically performs batch-by-batch abnormal data location, uses density-based clustering (DBSCAN) to identify abnormal data clusters, and outputs the location, size and possible root causes of abnormal data blocks to provide decision-making basis for manual review.

[0090] Furthermore, in step S6, the incremental online learning-driven optimization of the aggregation strategy and the iteration of the validation rules specifically include: Incremental status updates employ a sliding window mechanism, with different window sizes set for different business scenarios. Feature updates within the window use an exponentially weighted moving average. For example, for distribution network monitoring data with high real-time requirements, the window size... For load analysis data, Feature updates within the window use exponential weighting. The Moving Average (EWMA) formula is as follows:

[0091] in, The forgetting factor is adaptively set according to the business scenario; the strategy update adopts a gradient accumulation method, and the parameter update is performed once every K steps. The basic learning rate Based on business performance evaluations, the learning rate is dynamically adjusted using a Cyclic Learning Rate strategy to reduce the learning rate during critical business periods (peak electricity consumption times) to ensure decision stability.

[0092] Example 2 The architecture adopts a four-layer structure consisting of a data source layer, a data processing layer, a data service layer, and a data application layer. Each layer is decoupled and coordinated through standardized REST APIs and message queue interfaces. The data source layer is used for multi-source heterogeneous data acquisition and preprocessing to achieve standardized access to power data with different protocols and formats; The data processing layer is used for data cleaning, multi-scale feature extraction and intelligent fusion, and to complete intelligent data aggregation and full-process quality control. The data service layer is used to provide standardized data service interfaces and intelligent routing orchestration to realize unified publishing, dynamic discovery and on-demand access of aggregated data; The data application layer is used to achieve multi-business collaborative data application and value mining based on high-quality aggregated data, providing data-driven intelligent decision support for the entire power supply service process.

[0093] See Figure 2 As shown, this embodiment includes a multimodal data active perception and adaptive access module, a multi-scale feature extraction and representation learning module, a data aggregation decision optimization module, and a multi-level joint verification and feedback module; The multimodal data active perception and adaptive access module is used to collect multi-source heterogeneous power data, extract modal features of structured, time-series and unstructured data respectively, and generate a comprehensive representation of the data source through cross-modal attention mechanism; The multi-scale feature extraction and representation learning module is used to adaptively pool time-series data under multiple preset time scales, generate time-series multi-scale features through learnable scale attention fusion, extract distribution network topology spatial features using graph convolutional networks, and generate spatiotemporal joint representations through bilinear attention fusion of spatiotemporal features. The data aggregation decision optimization module is used to model the data aggregation problem as a Markov decision process, construct a system state vector containing spatiotemporal features, labels, and resource states; design a multi-dimensional composite reward function; use DuelingDoubleDQN combined with priority experience replay to train the decision agent; and dynamically output the optimal aggregation action through incremental online learning. The multi-level joint verification and feedback module is used to perform three-level progressive quality verification at the field level, record level, and dataset level; and to process abnormal data according to risk level.

[0094] The specific working principle is as follows: S1: Input multi-source heterogeneous data stream Initialize MDP parameters Initialize the parameters of the deep Q network and target network parameters .

[0095] S2: For each data source Through a three-branch feature recognition network Extracting comprehensive characterization Layered tags are obtained through a tag generation algorithm. .

[0096] S3: Based on spatiotemporal joint representation Constructing system state Selecting aggregation actions via DQN ( (Greedy strategy) Perform aggregation operation to obtain a new state .

[0097] S4: Perform M4 three-level verification (field level → record level → dataset level) and calculate the comprehensive reward. Transfer sample Store in the priority experience replay pool.

[0098] S5: When the replay pool size exceeds the threshold, sample a small batch of samples to update the DQN parameters. ; Each step synchronizes the target network C Implement an incremental online learning update strategy.

[0099] S6: Outputs high-quality aggregated data services, providing three levels of data service capabilities through standardized interfaces: real-time sharing of core business data, near-real-time sharing of supporting business data, and on-demand sharing of extended business data.

[0100] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0103] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0104] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A hierarchical, temporal, multi-scale learning-based intelligent aggregation method for multi-source heterogeneous data, characterized in that, Includes the following steps: Step S1: Extract features from structured data, time-series data, and unstructured data respectively, and fuse them through a cross-modal attention mechanism to obtain a comprehensive representation of the data source; Step S2: Based on the comprehensive characterization, a hierarchical tag generation framework is used to automatically generate basic data tags, quality assessment tags, and business scenario tags for the data source, thus forming a data source tag profile. Step S3: Perform adaptive pooling operations on the temporal features at M preset time scales to generate multi-scale representations. Obtain temporal multi-scale features by weighted fusion with learnable scale attention weights. Extract the distribution network topology spatial features using a graph convolutional network. Then, fuse the temporal multi-scale features with the spatial topology features through a bilinear attention mechanism to obtain a spatiotemporal joint representation. Step S4: Construct a mathematical model of the Markov decision process, build a state vector with spatiotemporal joint representation, data source label profile and system resource state, design a multi-dimensional composite reward function, and use dual-network deep Q-learning combined with priority experience replay to train the aggregation decision network and output the optimal aggregation action. Step S5: Perform field-level numerical verification, record-level business rule verification, and dataset-level global statistical verification in sequence, and process the detected abnormal data according to the risk level. Step S6: Combining the verification results and quality assessment report, the aggregation strategy is optimized and the verification rules are iterated through incremental online learning.

2. The method for intelligent aggregation of multi-source heterogeneous data using hierarchical temporal multi-scale learning according to claim 1, characterized in that: In step S1, features are extracted from structured data, time-series data, and unstructured data, specifically including: For the structured data branch, a multilayer perceptron with residual connections is used to extract numerical features, and a learnable embedding mapping is performed on the discrete category features. The numerical features and the embedded features are then concatenated to obtain the structured features. For the temporal data branch, a bidirectional long short-term memory network is used to capture long-term temporal dependencies. A multi-head temporal attention mechanism is introduced to extract key moment features. Three different one-dimensional convolutional layers with different kernel sizes are designed in parallel to extract multi-scale local temporal patterns. Attention features and convolutional features are concatenated to obtain temporal features. For unstructured data classification, a pre-trained BERT model is used to extract text semantic features, and ResNet-50 is used to extract image visual features. The text features and image features are then concatenated to obtain unstructured features.

3. The method for intelligent aggregation of multi-source heterogeneous data through hierarchical temporal multi-scale learning according to claim 1, characterized in that: In step S2, generating a framework through layered tags specifically includes: For the basic attribute label layer, a softmax classifier based on structured features outputs labels for data type, scale level, and update frequency. For the quality assessment label layer, the regression header of the time series features outputs scores for data integrity, reliability, and consistency. For the business scenario label layer, based on unstructured features, a graph neural network is used to construct semantic constraint relationships between labels and output the business scenario label probability. A three-layer label generation network is jointly trained using a multi-task learning loss function, where the loss function is: ； in, , , These are cross-entropy, mean squared error, and binary cross-entropy loss, respectively. This is a hyperparameter for task weights. It is the set of all learnable parameters.

4. The method for intelligent aggregation of multi-source heterogeneous data through hierarchical temporal multi-scale learning according to claim 1, characterized in that: In step S3, adaptive pooling operations are performed on the temporal features at M preset time scales to generate multi-scale representations. The temporal multi-scale features are then obtained through weighted fusion using learnable scale attention weights, specifically including: First, a set of time scales covering all scenarios of power business is predefined. S1 to SM correspond to time granularities of seconds, hours, days, and months, respectively, and M is the total number of time scales. Secondly, for the input raw time series data At each scale The pooling window size is automatically adjusted through adaptive pooling operations to generate a single-scale feature representation that matches the time scale, as follows: ； Secondly, for each single-scale feature representation Through the corresponding m-th scale feature extraction network Deep feature extraction is performed to obtain single-scale deep features. ; Secondly, all single-scale deep features are input into a learnable scale attention layer, and the attention weights corresponding to each time scale are calculated through linear transformation and softmax activation function. And all weights satisfy the normalization constraint. ; Finally, the weighted fusion of each single-scale depth feature with its corresponding attention weight is multiplied and summed to complete the weighted fusion of cross-scale features, resulting in the final temporal multi-scale fusion feature formula as follows: ； Among them, weight It is obtained through a learnable linear transformation plus softmax. is the feature extraction network at scale m.

5. The method for intelligent aggregation of multi-source heterogeneous data using hierarchical temporal multi-scale learning according to claim 1, characterized in that: In step S4, the data aggregation problem is formalized as a Markov decision process quintuple, including S, A, P, R, and Y. The specific definitions of each component are as follows: The state space S is characterized by spatiotemporal joint features. Label vectors Quality score vector Concatenate with system resource state vector constitute; Action space A includes aggregation operations that merge two data sources into the same aggregation group, separation operations that separate data sources from the current aggregation group, and refactoring operations that reorganize the global aggregation scheme according to changes in business scenarios. The reward function R is designed as a multi-dimensional composite function to comprehensively evaluate data quality, aggregation efficiency, semantic consistency, and resource consumption, guiding the agent to learn the optimal aggregation strategy. The specific formula is as follows: ； in, Greater than or equal to 0 and It equals 1 and is dynamically adjusted based on business priorities.

6. The method for intelligent aggregation of multi-source heterogeneous data using hierarchical temporal multi-scale learning according to claim 1, characterized in that: In step S4, a dual-network deep Q-learning combined with priority experience replay is used to train the aggregation decision network, specifically including: First, the spatiotemporal joint representation matrix output in step S3 and the adjacency matrix of the current aggregation state are input into the graph-integrated convolutional network to extract the structured state code that simultaneously contains the spatiotemporal features of the data and the aggregation structure information. Secondly, the structured state code is input into the Dueling backbone network and decomposed into state values. and dominance function The overall value of the current state and the relative advantage of each aggregate action are output separately, and the final Q value is obtained through combination calculation. The specific formula is as follows: ； Secondly, set up online networks with completely identical structures. and target network The online network is responsible for selecting the optimal aggregation action in real time, while the target network is responsible for calculating a stable target Q value, thus separating the action selection and target Q value calculation processes. Finally, an experience replay pool is constructed to store transfer samples of agent-environment interaction. Different sampling priorities are assigned to samples according to the magnitude of TD error, important samples are given higher sampling weights, and sampling bias is corrected through importance sampling.

7. The method for intelligent aggregation of multi-source heterogeneous data through hierarchical temporal multi-scale learning according to claim 1, characterized in that: Step S5 specifically includes: The field-level numerical verification employs an improved interquartile range (ICM) method for adaptive detection of numerical anomalies. The detection coefficient k is dynamically adjusted based on the kurtosis of the data distribution, as shown in the following formula: Combined with isolated forest detection of high-dimensional feature anomalies; The record-level business rule verification checks the cross-field logical relationship and temporal continuity of data records, and uses a multi-classifier integration to identify error types; The dataset-level global statistical verification is based on a statistical process control framework. It uses an LSTM time-series anomaly detection model to monitor changes in key statistics of the dataset and identifies anomalous data clusters through DBSCAN clustering.

8. The method for intelligent aggregation of multi-source heterogeneous data through hierarchical temporal multi-scale learning according to claim 1, characterized in that: In step S6, the incremental online learning-driven optimization of the aggregation strategy and iteration of the verification rules specifically include: Incremental state updates employ a sliding window mechanism, with different window sizes set for different business scenarios. Feature updates within the window utilize an exponentially weighted moving average, as shown in the following formula: ； in, The forgetting factor is adaptively set according to the business scenario; the strategy update adopts a gradient accumulation method, and the parameter update is performed once every K steps. The base learning rate is dynamically adjusted according to the business effect evaluation.

9. A hierarchical temporal multi-scale learning intelligent aggregation system for multi-source heterogeneous data, characterized in that: It includes a multimodal data active perception and adaptive access module, a multi-scale feature extraction and representation learning module, a data aggregation decision optimization module, and a multi-level joint verification and feedback module; The multimodal data active perception and adaptive access module is used to collect multi-source heterogeneous power data, extract modal features of structured, time-series and unstructured data respectively, and generate a comprehensive representation of the data source through cross-modal attention mechanism; The multi-scale feature extraction and representation learning module is used to adaptively pool time-series data under multiple preset time scales, generate time-series multi-scale features through learnable scale attention fusion, and extract distribution network topology features using graph convolutional networks. Spatiotemporal features are fused using bilinear attention to generate a joint spatiotemporal representation; The data aggregation decision optimization module is used to model the data aggregation problem as a Markov decision process, construct a system state vector containing spatiotemporal features, labels, and resource states; design a multi-dimensional composite reward function; use DuelingDoubleDQN combined with priority experience replay to train the decision agent; and dynamically output the optimal aggregation action through incremental online learning. The multi-level joint verification and feedback module is used to perform three-level progressive quality verification at the field level, record level, and dataset level. Abnormal data is classified and processed according to its risk level.

Images