Multi-target library real-time data synchronization method and system for electric power metering

Abstract

The application discloses a multi-target library real-time data synchronization method and system for electric power metering, and belongs to the technical field of electric power metering data processing. An edge node device collects real-time data of electric power metering terminals and pre-processes, and uploads a state matrix of energy consumption and resources to the cloud after generation; the cloud automatically detects target library types, determines adaptive priorities through dynamic format conversion and analytic hierarchy process; a five-dimensional target function containing business priority, cross-library adaptive success rate, system energy consumption, network load balancing and data time sequence consistency is constructed based on the state matrix and the priority, and an optimal scheduling scheme is generated through a DDPG algorithm; transmission is completed through LSTM prediction, sliding window difference and lightweight compression, and a hash value and a device ID are written into an electric power industry alliance chain after SHA-256 segment verification. The application realizes multi-library collaborative adaptation, low bandwidth occupation, low energy consumption and high time sequence consistency, and guarantees real-time, accurate and safe metering data synchronization.

Description

Technical Field

[0001] This invention relates to the field of power metering data processing technology, and in particular to a method and system for real-time data synchronization of multi-target databases for power metering. Background Technology

[0002] Currently, electricity metering data serves as the core data support for power grid dispatching strategies, accurate electricity billing, and optimization of electricity services. Its real-time performance, accuracy, and security directly affect the stable operation of the power system and the operating efficiency of power companies.

[0003] With the deepening of smart grid construction, the number of metering terminals has exploded, and the acquisition frequency of a single terminal has increased to 1Hz. The resulting metering data is characterized by strong periodicity, burstiness, high accuracy requirements, and heterogeneous storage across multiple systems, which traditional data synchronization technologies can no longer meet. Existing power data synchronization technologies have a single optimization objective, rigid synchronization mechanisms, and fail to utilize the temporal regularity of power data, resulting in serious waste of network bandwidth, poor cross-database compatibility, and high adaptation costs.

[0004] Therefore, it is necessary to improve existing technologies. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for real-time data synchronization of multiple target databases for power metering, aiming to solve the technical problems in the prior art, such as single target optimization, rigid synchronization mechanism, failure to utilize the temporal regularity of power data, resulting in serious waste of network bandwidth, poor cross-database compatibility, and high adaptation cost.

[0006] To achieve the objectives of this invention, the specific technical solution provided by this invention is as follows: First aspect This invention provides a method for real-time data synchronization of multi-target databases for power metering, comprising the following steps: Step 1: Use edge node devices to collect real-time data from power metering terminals, preprocess the real-time data from power metering terminals to obtain the energy consumption and resource status matrix and the preprocessed data; upload the energy consumption and resource status matrix and the preprocessed data to the cloud server together. Step 2: After receiving the preprocessed data, the cloud server automatically detects the target library type and extracts the storage format and field constraint features; it uses a dynamic format conversion model based on abstract syntax tree to adapt the format of the preprocessed data, and uses the hierarchical analysis method to determine the target library adaptation priority, forming an adapted data and adaptation priority list. Step 3: Construct a five-dimensional objective function based on the energy consumption and resource state matrix and the adaptation priority list, and use the DDPG reinforcement learning algorithm to generate the optimal scheduling scheme; wherein, the five-dimensional objective function includes business priority objective, cross-database adaptation success rate objective, system energy consumption objective, network load balancing objective and data time sequence consistency objective; Step 4: Based on the optimal scheduling scheme, first use the LSTM model to predict the future 5-minute change trend of the adapted data, and only use the sliding window difference algorithm to process the data that is predicted to change. Combine the target library type to select a lightweight compression algorithm and a dynamic transmission path to complete the data transmission. Step 5: Use the SHA-256 hash algorithm to perform segmented hash verification of the transmitted data in real time. After synchronization is completed, write the verified data hash value and edge node device ID into the power industry consortium blockchain.

[0007] Second aspect This invention also provides a real-time data synchronization system for a multi-target database for power metering, comprising edge node devices and a cloud server; the edge node devices and the cloud server are used to perform the following steps: Step 1: Use edge node devices to collect real-time data from power metering terminals, preprocess the real-time data from power metering terminals to obtain the energy consumption and resource status matrix and the preprocessed data; upload the energy consumption and resource status matrix and the preprocessed data to the cloud server together. Step 2: After receiving the preprocessed data, the cloud server automatically detects the target library type and extracts the storage format and field constraint features; it uses a dynamic format conversion model based on abstract syntax tree to adapt the format of the preprocessed data, and uses the hierarchical analysis method to determine the target library adaptation priority, forming an adapted data and adaptation priority list. Step 3: Construct a five-dimensional objective function based on the energy consumption and resource state matrix and the adaptation priority list, and use the DDPG reinforcement learning algorithm to generate the optimal scheduling scheme; wherein, the five-dimensional objective function includes business priority objective, cross-database adaptation success rate objective, system energy consumption objective, network load balancing objective and data time sequence consistency objective; Step 4: Based on the optimal scheduling scheme, first use the LSTM model to predict the future 5-minute change trend of the adapted data, and only use the sliding window difference algorithm to process the data that is predicted to change. Combine the target library type to select a lightweight compression algorithm and a dynamic transmission path to complete the data transmission. Step 5: Use the SHA-256 hash algorithm to perform segmented hash verification of the transmitted data in real time. After synchronization is completed, write the verified data hash value and edge node device ID into the power industry consortium blockchain.

[0008] Compared with the prior art, the beneficial effects of the present invention are as follows: The technical solution provided by this invention achieves coordinated optimization of business priority, cross-database adaptation, energy consumption, network load, and timing consistency through the constructed five-dimensional objective function and DDPG reinforcement learning algorithm. It breaks through the limitation of a single objective, realizes multi-dimensional coordinated optimization, and can adapt to the complex needs of smart grids. In addition, this invention constructs an automated and dynamic cross-database adaptation system, which can automatically detect database types and extract features such as field precision and storage structure through JDBC interfaces / RESTful APIs without manual intervention; based on the abstract syntax tree, it parses the target database syntax rules and combines them with a preset power data type mapping table to achieve automatic conversion from a unified JSON format to the native syntax of each database. When adaptation fails, it can automatically adjust the conversion rules through closed-loop optimization, avoiding repeated development and significantly reducing the adaptation cost and maintenance difficulty in multi-database heterogeneous scenarios. In addition, this invention also solves the problems of poor cross-database compatibility, bandwidth waste, and high energy consumption in traditional technologies, realizes multi-objective collaborative optimization, and ensures the real-time, accuracy and security of power metering data, which can provide reliable data support for power grid dispatching and accurate billing. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of a multi-target database real-time data synchronization method for power metering provided in an embodiment of this application. Detailed Implementation

[0010] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0011] like Figure 1 As shown in the figure, this embodiment provides a method for real-time data synchronization of multi-target databases for power metering, including the following steps: Step 1: Use edge node devices to collect real-time data from power metering terminals, preprocess the real-time data from power metering terminals to obtain the energy consumption and resource status matrix and the preprocessed data; upload the energy consumption and resource status matrix and the preprocessed data to the cloud server together. Step 1 specifically includes the following: Step 1.1: Use edge node devices to collect real-time data from power metering terminals; Step 1.2: Remove extreme outliers using the Isolation Forest algorithm, which incorporates temporal characteristics; Step 1.3: After removing extreme outliers, the valid data is distinguished into steady-state data and burst data. The steady-state data is aggregated by minute-level average, and the voltage fluctuation of the steady-state data is ≤0.5%. The burst data is retained as the original data to obtain the preprocessed data. The node energy consumption and network resource status are monitored simultaneously to generate an energy consumption and resource status matrix. Step 1.4: Upload the preprocessed data and the energy consumption and resource status matrix to the cloud server.

[0012] It should be noted that the edge node device is an ARM Cortex-A72 architecture embedded device, which uses a power-specific RS485 / 5G module to connect to the metering terminal. A single node supports 80-100 terminals to connect, and the preprocessing latency is ≤50ms. It should be noted that the energy consumption and resource state matrix is ​​a two-dimensional data containing energy consumption and bandwidth, generated by collecting CPU power consumption through the power consumption sensor built into the edge node device and bandwidth utilization through the network monitoring module.

[0013] It should be noted that the specific implementation method of the isolated forest algorithm is as follows: The time-series change rate of electricity data is introduced as a feature dimension; an analysis unit is constructed using a 5-minute sliding window; the mean and standard deviation of the data within the window are calculated; and data deviating from 3... Extreme outliers, with an anomaly detection accuracy of ≥98%; Step 2: After receiving the preprocessed data, the cloud server automatically detects the target library type and extracts the storage format and field constraint features; it uses a dynamic format conversion model based on abstract syntax tree to adapt the format of the preprocessed data, and uses the hierarchical analysis method to determine the target library adaptation priority, forming an adapted data and adaptation priority list. The target library types include MySQL, InfluxDB, MongoDB, and Oracle.

[0014] It should be noted that in step 2, the hierarchical analysis method is used to determine the target library adaptation priority, as follows: A three-tiered indicator system is constructed, which includes three indicators: business importance (weight 0.6), storage capacity (weight 0.2), and write performance (weight 0.2). Among them, the business importance indicator is calculated by weighting the billing system (weight 0.4), scheduling system (weight 0.3), operation and maintenance system (weight 0.2), and backup database (weight 0.1). A priority list for adaptation is generated by combining the scores of various indicators.

[0015] It should be noted that step 2, the adaptation method includes the following steps: By probing relational databases through JDBC interfaces and non-relational databases through RESTful APIs, the field precision and storage structure characteristics of the target databases can be extracted. The dynamic format conversion model parses the target library's syntax rules based on an abstract syntax tree, converting the unified JSON format data output by the edge nodes into the native syntax of the corresponding library; Type mapping is achieved through a pre-defined mapping table between power data types and target library fields.

[0016] Step 3: Construct a five-dimensional objective function based on the energy consumption and resource state matrix and the adaptation priority list, and use the DDPG reinforcement learning algorithm to generate the optimal scheduling scheme; wherein, the five-dimensional objective function includes business priority objective, cross-database adaptation success rate objective, system energy consumption objective, network load balancing objective and data time sequence consistency objective; It should be noted that the five-dimensional objective function in step 3 specifically includes the following: (1) Business priority objectives Weights are assigned based on billing and scheduling services. Synchronization timeliness is judged by ≤1s, with 1 being compliant and 0 being non-compliant. in, In the formula, i=1 represents billing services, i=2 represents scheduling services, i=3 represents operation and maintenance services, i=4 represents backup services, and W i As a business weight, For synchronization and timeliness; (2) Cross-database adaptation success rate target Calculated based on the adaptation results of the adapted data; in, =Amount of data successfully written to the target database / Total amount of synchronized data; (3) System energy consumption target It is calculated by summing the power-time products of the edge node devices and the cloud server; Among them, the power parameter is the monitoring data of edge node devices and cloud servers, the goal is to minimize total energy consumption, the power is the average power over 1 minute in W, and the time granularity is the duration of the synchronization task in h. (4) Network load balancing objectives The calculation method is as follows: =1-[(Maximum bandwidth utilization - Minimum bandwidth utilization) / Average bandwidth utilization]; Among them, the bandwidth utilization rate data is collected by edge node devices, with a target value ≥0.85; (5) Data time sequence consistency target The core principle is to ensure that the timestamps of synchronized data are unbroken. in, =1-(Number of timestamp breaks after synchronization / Total number of data entries), where timestamp break is defined as: the interval between adjacent data timestamps exceeds the collection period; (6) Reward function The calculation method is as follows: .

[0017] Step 4: Based on the optimal scheduling scheme, first use the LSTM model to predict the future 5-minute change trend of the adapted data, and only use the sliding window difference algorithm to process the data that is predicted to change. Combine the target library type to select a lightweight compression algorithm and a dynamic transmission path to complete the data transmission. It should be noted that in step 4, the prediction of the future 5-minute change trend of the adapted data using the LSTM model is specifically as follows: using voltage and current time series data sampled at 1Hz over the past hour as the training set, the Adam optimizer is used for training with a training rate of 0.001, and the future 5-minute data change trend is output. The sliding window difference algorithm is used for processing, specifically: the sliding window size is set to 100 data points, the difference between the current data and the previous data in the window is calculated, and only the difference is transmitted. The lightweight compression algorithm is selected as follows: for time-series databases, the LZ77 algorithm is used, which achieves compression by matching repeating byte sequences through a sliding window; for non-relational databases, the Snappy algorithm is used, which is based on the fast compression logic of a variant of LZ77; for relational databases, the Gzip algorithm is used, which combines Huffman coding to optimize the compression effect; the compression ratio of each algorithm is ≥1:5, and the compression / decompression latency is ≤10ms. The method for selecting the dynamic transmission path is as follows: based on the bandwidth idle rate determination in the energy consumption and resource state matrix, high-priority tasks adopt a dual-path mechanism with dedicated power lines as the main path and 5G as the backup path, and switch in real time through the link quality detection module; low-priority tasks adopt public network transmission, and avoid bandwidth occupation conflicts through traffic control strategies.

[0018] Step 5: Use the SHA-256 hash algorithm to perform segmented hash verification of the transmitted data in real time. After synchronization is completed, write the verified data hash value and edge node device ID into the power industry consortium blockchain.

[0019] It should be noted that step 5 specifically includes the following: The SHA-256 hash algorithm is used to perform segmented hash real-time verification of transmitted data. Each 1KB of data is transmitted triggers an SHA-256 hash calculation, which is compared with the hash value pre-stored by the edge node device in real time through point-to-point comparison. If an inconsistency is found, the transmission is immediately interrupted and retransmission is initiated. After synchronization is complete, the verified data hash value, edge node device ID, and timestamp information are encapsulated into a transaction and sent to the power industry consortium chain node composed of power companies, metering manufacturers, and regulatory agencies. After verification by the PBFT consensus algorithm, the transaction is written into a block, with a block generation time of ≤3s.

[0020] It should be noted that the inputs to the PBFT consensus algorithm are: a hash value that has passed SHA-256 verification; core metadata (synchronization timestamp, edge node ID, target library identifier, verification result); and consortium blockchain node signatures (power company / manufacturer / regulatory agency); the outputs are: blockchain transaction records (encapsulated metadata + hash); and block write confirmations (including block height and consensus time). Preferably, step 5 further includes a step of achieving closed-loop optimization of the entire process through a feedback mechanism: When adaptation fails, the conversion rules of the abstract syntax tree are automatically adjusted by analyzing the fields with syntax mismatch through log analysis. When the prediction error is ≥10%, the latest actual measurement data is added to the training set of the LSTM model and the iteration is restarted. When energy consumption exceeds the limit, the scheduling algorithm reduces the CPU allocation ratio of non-core tasks on edge nodes to below 20%, dynamically balancing energy consumption and performance.

[0021] It should be noted that the DDPG reinforcement learning algorithm framework described in this embodiment includes a state preprocessing module, an Actor network, a Critic network, a priority experience replay pool, a multi-objective reward fusion module, and an adaptive exploration module. The state preprocessing module receives 12-dimensional raw state data, performs normalization and anomaly filtering, and outputs a standardized state vector, as follows: Input: 12-dimensional raw state data (edge ​​energy consumption 2-dimensional: CPU power consumption, bandwidth utilization; target library state 3-dimensional: load rate, remaining capacity, write TPS; network state 2-dimensional: bandwidth idle rate, link reliability; service type 5-dimensional: 4 types of service priorities + data type identifier).

[0022] Processing logic: Normalization: Min-max normalization is used to map the data of each dimension to the [0,1] interval; Anomaly filtering: Deviations in the state data are removed. For extreme values, fill with the mean of the previous time step; Output: A 12-dimensional normalized state vector S (dimension: 1×12).

[0023] The Actor network outputs the optimal scheduling action based on the standardized state vector, specifically including the following: Network structure: 3 fully connected layers + BatchNorm layer + LeakyReLU activation Input layer: 12-dimensional state vector S; Hidden layer 1: 256 neurons, BatchNorm layer + LeakyReLU (α=0.01); Hidden layer 2: 128 neurons, BatchNorm layer + LeakyReLU (α=0.01); Output layer: 6-dimensional action vector A (dimension: 1×6), activation function is Sigmoid (mapped to the [0,1] interval and then decoupled); Key parameters: Learning rate: Initially 0.0001, decaying to 0.00005 after 300 iterations (using an exponential decay strategy). Attenuation coefficient ); Weight initialization: He normal distribution (adapted LeakyReLU activation) The target network update rate τ = 0.005 (soft update: Target_Actor = τ × Actor + (1-τ) × Target_Actor).

[0024] The Critic network combines state vectors and action vectors to evaluate the comprehensive benefits of the scheduling scheme across five dimensions; specifically, it includes the following: Network structure: Dual Critic network, each layer is fully connected + BatchNorm layer + LeakyReLU activation; Input layer: 12-dimensional state vector S + 6-dimensional action vector A (concatenated to form an 18-dimensional input); Hidden layer 1: 256 neurons, BatchNorm layer + LeakyReLU (α=0.01); Hidden layer 2: 128 neurons, BatchNorm layer + LeakyReLU (α=0.01); Output layer: 1-dimensional value evaluation value Q (representing the combined reward of the current state-action pair). Key parameters: Learning rate: 0.0002 (higher than Actor networks to ensure accurate value assessment); Loss function: MSE loss (… The target network update rate τ = 0.005 (consistent with the Actor network). The priority experience replay pool stores experience groups, which are sampled according to priority for model updates; specifically, it includes the following: Pool capacity: 1 experience group (each experience group format: ( )); Priority Calculation: Priority , =0.01, avoid priority of 0, TD_error=Q_target-Q_pred; Sampling strategy: By Probability sampling ( =0.6, initial weight; linearly increased to 1.0 during iteration to balance priority and randomness). Experience grouping: Divided into 4 sub-pools according to business type (billing / scheduling / operation / backup), each sub-pool has a weighting ratio of 4:3:2:1 to ensure that high-priority business experience is fully learned.

[0025] The adaptive exploration module dynamically adjusts motion noise to balance exploration and utilization; specifically, it includes the following: Noise type: Ornstein-Uhlenbeck (OU) noise (simulating dynamic fluctuation characteristics of power systems); Dynamic adjustment rules for noise parameters: Initial noise variance 0 = 0.2; Calculate the overall achievement rate of the five-dimensional objectives every 10 iterations. ; If R ≥ 0.9, = ×0.95 (noise attenuation); If R < 0.7, = ×1.05 (Enhanced Exploration); Noise lower limit min =0.01 (to avoid completely stopping the exploration).

[0026] Key training parameters: Total number of iterations: 500; Batch size: 256 experience points / batch; Discount factor γ: 0.95; Convergence criteria: Q value passively ≤1% for 20 consecutive iteration batches, and the overall achievement rate of the five-dimensional objectives R ≥ 0.92; Bandwidth idle rate = (total link bandwidth - real-time occupied bandwidth) / total link bandwidth × 100%; When the bandwidth idle rate is ≥ 60%, the dedicated power line is used first; When the bandwidth idle rate is < 60%, it automatically switches to the 5G backup link; It should be noted that the input to the abstract syntax tree is: unified JSON format data after edge preprocessing; target library syntax rules (from feature recognition in step 2, such as MySQL's INSERT syntax and InfluxDB's LineProtocol); power data type mapping table (decimal→MySQLDECIMAL, etc.); the output is: target library native format data (such as SQL statements and BSON documents); type mapping log (matching results and correction records).

[0027] Test case This test case is applied to a provincial smart grid metering data management system. The system covers the urban power distribution networks of 10 prefecture-level cities, connecting 8,000 power metering terminals. It needs to achieve real-time synchronization of metering data to multiple target databases. Specifically, MySQL is used to store billing data, InfluxDB to store time-series operational monitoring data, MongoDB to store terminal maintenance logs, and Oracle to store backup data and core business files.

[0028] The core requirements of the system are: a single terminal sampling frequency of 1Hz, generating approximately 86,400 data entries per day, requiring real-time synchronization across multiple databases; billing and scheduling services require synchronization timeliness of ≤1s, while maintenance and backup services can be moderately relaxed; reduce energy consumption for transmission and processing between edge nodes and the cloud to avoid network bandwidth congestion; ensure cross-database adaptation and compatibility, uninterrupted data sequence, and traceability of the synchronization process.

[0029] This test case uses the technical solution of the present invention, and the hardware configuration is as follows: Edge node device: ARM Cortex-A72 architecture embedded device (model: Raspberry Pi 4B Enhanced Edition), equipped with a power-specific RS485 / 5G dual-mode module (5G band: Sub-6GHz, RS485 communication rate: 9600bps), a single node can connect 90 metering terminals, and has a built-in power consumption sensor (measurement range: 0-10W, accuracy ±0.1W) and network monitoring module.

[0030] Cloud servers: 2 eight-core 16G cloud servers (CPU: Intel Xeon E5-2680v4, memory: 16GB DDR4, bandwidth: 100Mbps), deploying target library cluster and algorithm scheduling module.

[0031] Consortium blockchain nodes: one node each from the power company, metering equipment manufacturer, and energy regulatory agency, using the PBFT consensus mechanism, with a block storage capacity of 1TB.

[0032] The execution process is as follows: (1) Data acquisition and preprocessing Data Acquisition: The edge node device connects to 90 power metering terminals via RS485 modules to collect real-time data such as voltage (range: 0-400V), current (range: 0-100A), active power (range: 0-40kW), and reactive power. The sampling frequency is 1Hz, and the data format is raw binary stream.

[0033] Outlier removal: An improved isolated forest algorithm was used, with an analysis unit constructed using a 5-minute sliding window (containing 300 data points). The mean voltage data within the window was calculated to be 220V with a standard deviation of 0.3V. Extreme outliers deviating from 3σ (i.e., >220.9V or <219.1V) were removed. In this batch of 300 data points, two outliers with a voltage value of 221.5V were detected and successfully removed, achieving an outlier detection accuracy of 99.3% (meeting the requirement of ≥98%).

[0034] Data differentiation and processing: After removing anomalies, the 298 valid data points were judged. Among them, 280 data points had voltage fluctuations ≤0.5% (belonging to steady-state data), and minute-level average aggregation was performed to obtain 5 minute-level average data points; 18 data points had voltage fluctuations >0.5% (belonging to burst data, such as the moment of terminal start-up and shutdown), and the original data were retained.

[0035] State matrix generation: The CPU power consumption was collected as 3.2W (average power over 1 minute) by the node's built-in power sensor, and the bandwidth utilization rate was collected as 45% by the network monitoring module. The state matrix was generated as: [[3.2,45%]].

[0036] Data upload: The 5 preprocessed aggregated data points, 18 original data points (a total of 23 preprocessed data points), and the state matrix are uploaded to the cloud together. The preprocessing latency is 38ms (which meets the requirement of ≤50ms).

[0037] (2) Cloud target library adaptation and priority sorting Target database type detection: MySQL and Oracle (relational databases) are detected via JDBC interface in the cloud, and InfluxDB (time-series database) and MongoDB (document database) are detected via RESTful API. Characteristics of each database are extracted: MySQL fields have a precision of decimal(10,2) and a table structure; InfluxDB supports LineProtocol format and stores data in partitions by timestamp; MongoDB uses BSON document structure and fields have no strict constraints; Oracle fields have a precision of number(12,3) and support transactions.

[0038] Format adaptation: Based on the abstract syntax tree, the syntax rules of each target library are parsed, and the unified JSON format data output by the edge nodes is converted into native syntax: MySQL corresponds to INSERT statements, InfluxDB corresponds to the format: power_monitor,terminal_id=001 voltage=220.3,current=5.2,power=1144.6 1699876543, MongoDB corresponds to document objects, and Oracle corresponds to INSERT statements. Type mapping is implemented through a preset mapping table.

[0039] Priority determination: A three-level indicator system is constructed using the analytic hierarchy process (AHP). Primary indicator weights: Business importance 0.6, storage capacity 0.2, write performance 0.2; Secondary weighting of business importance: Billing system 0.4, Dispatch system 0.3, Operation and maintenance system 0.2, Backup database 0.1; Score calculation for each target library: MySQL (Billing Service) Score = 0.6 × 0.4 + 0.2 × 0.8 (Sufficient Storage Capacity) + 0.2 × 0.9 (High Write TPS) = 0.46; InfluxDB (scheduling service) score = 0.6 × 0.3 + 0.2 × 0.7 + 0.2 × 0.95 = 0.38; MongoDB (Operations and Maintenance) score = 0.6 × 0.2 + 0.2 × 0.6 + 0.2 × 0.8 = 0.28; Oracle (backup service) score = 0.6 × 0.1 + 0.2 × 0.9 + 0.2 × 0.7 = 0.20; Generate a priority list: MySQL (Level 1) > InfluxDB (Level 2) > MongoDB (Level 3) > Oracle (Level 4).

[0040] (3) Construction of five-dimensional objective function and generation of optimal scheduling scheme Five-dimensional objective function calculation: Business priority goals ( Synchronization timeliness of design fee (i=1), scheduling (i=2), operation and maintenance (i=3), and backup (i=4): T1=1 (≤1s meets the standard), T2=1, T3=1, T4=1, weights W1=0.4, W2=0.3, W3=0.2, W4=0.1. =0.4×1+0.3×1+0.2×1+0.1×1=1.0; Cross-database adaptation success rate target ( Total synchronized data volume: 23 records; 23 records were successfully written to 4 target databases. =23 / 23=1.0; Energy consumption target ( Edge node power 3.2W, synchronization task duration 0.01h (36s), energy consumption = 3.2 × 0.01 = 0.032Wh; cloud server power 80W, synchronization task duration 0.005h (18s), energy consumption = 80 × 0.005 = 0.4Wh; total energy consumption = 0.032 + 0.4 = 0.432Wh (target minimization). Network load balancing objectives ( The maximum bandwidth utilization rate is 55%, the minimum is 35%, and the average is 45%. =1-[(55%-35%) / 45%]=1-0.444=0.556 (subsequently improved to ≥0.85 through scheduling optimization); Data time series consistency goal ( Total data entries: 23; Number of timestamp breaks: 0. =1-(0 / 23)=1.0; The reward function R = 0.3 × 1.0 + 0.25 × 1.0 + 0.2 × (1 - 0.432 / 1) (after energy consumption normalization) + 0.15 × 0.556 + 0.1 × 1.0 ≈ 0.3 + 0.25 + 0.1136 + 0.0834 + 0.1 = 0.847.

[0041] DDPG algorithm scheduling: State input: 12-dimensional normalized state vector (edge ​​power consumption: [3.2W, 45%] → [0.32, 0.45]; target library state: [load rate 30%, remaining capacity 80%, write TPS 1000] → [0.3, 0.8, 0.5]; network state: [bandwidth idle rate 55%, link reliability 99%] → [0.55, 0.99]; service type: [0.4, 0.3, 0.2, 0.1, 0.1]). Model training: 500 iterations, batch size of 256 experience points, Actor network learning rate decayed from 0.0001 to 0.00005, Critic network learning rate of 0.0002, converged after 350 iterations (Q value fluctuation ≤1% for 20 consecutive batches, R≥0.92). Optimal scheduling scheme output: Transmission order is MySQL→InfluxDB→MongoDB→Oracle; Edge node CPU allocation ratio: 80% for core tasks (data preprocessing), 15% for non-core tasks; Cloud CPU allocation ratio: 30% for MySQL writes, 25% for InfluxDB writes, 20% for MongoDB writes, and 15% for Oracle writes; Synchronization frequency: 2Hz for MySQL and InfluxDB, and 0.5Hz for MongoDB and Oracle; Transmission path: MySQL and InfluxDB (high priority) use dedicated power lines (automatic switch to 5G backup link when bandwidth idle rate is 55% < 60%), while MongoDB and Oracle (low priority) use public network transmission.

[0042] (4) Data transmission optimization LSTM prediction: Using the past hour (3600 data points) of voltage and current time series data as the training set, the LSTM model was trained with the Adam optimizer (learning rate 0.001) to predict the data change trend in the next 5 minutes: voltage fluctuation ≤0.3%, current fluctuation ≤0.2%, 18 data points were judged as "no significant change", and 5 data points were judged as "change".

[0043] Sliding window differential processing: For 5 data points that have changed, a sliding window differential algorithm with a window size of 100 lines is used to calculate the difference between the current data and the previous data in the window (e.g., the current voltage is 220.5V, the previous average is 220.2V, and the difference is 0.3V). Only the difference part is transmitted, reducing the amount of data by 85%.

[0044] Lightweight compression: The algorithm is selected according to the target database type: InfluxDB (time series database) uses the LZ77 algorithm with a compression ratio of 1:5.2; MySQL and Oracle (relational databases) use the Gzip algorithm with a compression ratio of 1:5.8; MongoDB (non-relational database) uses the Snappy algorithm with a compression ratio of 1:5.1; the compression latency is ≤8ms (meeting the ≤10ms requirement).

[0045] Dynamic transmission: High-priority tasks (MySQL, InfluxDB) are transmitted via 5G links (bandwidth idle rate 55%), and the link quality detection module monitors the packet loss rate in real time to be <0.1%; low-priority tasks (MongoDB, Oracle) are transmitted via the public network, and the traffic control strategy limits the bandwidth utilization rate to ≤30% to avoid conflicts.

[0046] (5) Data verification and closed-loop optimization SHA-256 segmented hash verification: The compressed data is divided into 1KB segments (3 segments in total). The hash value of each segment is calculated and compared point-to-point with the pre-stored hash value of the edge node. If all 3 segments are consistent, no retransmission is triggered.

[0047] Blockchain Evidence Storage: After synchronization is completed, the data hash, timestamp (1699876579), and edge node ID (ED-001) are encapsulated into a transaction and sent to 3 nodes of the consortium blockchain. After the PBFT consensus algorithm verifies the transaction, it is written into the block. The block generation time is 2.3 seconds (meeting the ≤3 seconds requirement).

[0048] Closed-loop optimization: Adaptation and optimization: Simulate MySQL field precision mismatch (original data decimal(10,3) vs target database decimal(10,2)), locate mismatched fields through log analysis, automatically adjust the abstract syntax tree conversion rules (keeping 2 decimal places), and subsequently improve the adaptation success rate to 100%; Model update: The LSTM predicted current data with a deviation of 12% (≥10%) for a certain period. After adding 100 actual measurement data points for that period to the training set and iterating again, the prediction deviation was reduced to 7%. Energy consumption optimization: The CPU allocation ratio of non-core tasks on edge nodes was detected to be 25% (exceeding the standard). The scheduling algorithm reduced it to 18% (<20%), and the total energy consumption was reduced to 0.38Wh, a reduction of 12%.

[0049] Implementation effect verification: This test case achieved real-time synchronization of power metering data to multiple target databases through the above steps; anomaly detection accuracy: 99.3%; preprocessing latency: 38ms; cross-database adaptation success rate: 100%; business synchronization timeliness (billing / scheduling): ≤0.8s; network load balancing target value: 0.88; data timing consistency: 100%; compression ratio: 1:5.1-1:5.8; block generation time: 2.3s; total energy consumption reduction rate: 12% (after iteration).

[0050] The present invention solves the problems of poor cross-database compatibility, bandwidth waste, and high energy consumption in traditional technologies, realizes multi-objective collaborative optimization, and ensures the real-time, accuracy and security of power metering data, providing reliable data support for power grid dispatching and accurate billing.

[0051] In addition, this embodiment provides a real-time data synchronization system for a multi-target database for power metering, including edge node devices and a cloud server; the edge node devices and the cloud server are used to perform the following steps: Step 1: Use edge node devices to collect real-time data from power metering terminals, preprocess the real-time data from power metering terminals to obtain the energy consumption and resource status matrix and the preprocessed data; upload the energy consumption and resource status matrix and the preprocessed data to the cloud server together. Step 2: After receiving the preprocessed data, the cloud server automatically detects the target library type and extracts the storage format and field constraint features; it uses a dynamic format conversion model based on abstract syntax tree to adapt the format of the preprocessed data, and uses the hierarchical analysis method to determine the target library adaptation priority, forming an adapted data and adaptation priority list. Step 3: Construct a five-dimensional objective function based on the energy consumption and resource state matrix and the adaptation priority list, and use the DDPG reinforcement learning algorithm to generate the optimal scheduling scheme; wherein, the five-dimensional objective function includes business priority objective, cross-database adaptation success rate objective, system energy consumption objective, network load balancing objective and data time sequence consistency objective; Step 4: Based on the optimal scheduling scheme, first use the LSTM model to predict the future 5-minute change trend of the adapted data, and only use the sliding window difference algorithm to process the data that is predicted to change. Combine the target library type to select a lightweight compression algorithm and a dynamic transmission path to complete the data transmission. Step 5: Use the SHA-256 hash algorithm to perform segmented hash verification of the transmitted data in real time. After synchronization is completed, write the verified data hash value and edge node device ID into the power industry consortium blockchain.

[0052] In step 3, the five-dimensional objective function specifically includes the following: (1) Business priority objectives Weights are assigned based on billing and scheduling services. Synchronization timeliness is judged by ≤1s, with 1 being compliant and 0 being non-compliant. in, In the formula, i=1 represents billing services, i=2 represents scheduling services, i=3 represents operation and maintenance services, i=4 represents backup services, and W i As a business weight, For synchronization and timeliness; (2) Cross-database adaptation success rate target Calculated based on the adaptation results of the adapted data; in, =Amount of data successfully written to the target database / Total amount of synchronized data; (3) System energy consumption target It is calculated by summing the power-time products of the edge node devices and the cloud server; Among them, the power parameter is the monitoring data of edge node devices and cloud servers, the goal is to minimize total energy consumption, the power is the average power over 1 minute in W, and the time granularity is the duration of the synchronization task in h. (4) Network load balancing objectives The calculation method is as follows: =1-[(Maximum bandwidth utilization - Minimum bandwidth utilization) / Average bandwidth utilization]; Among them, the bandwidth utilization rate data is collected by edge node devices, with a target value ≥0.85; (5) Data time sequence consistency target The core principle is to ensure that the timestamps of synchronized data are unbroken. in, =1-(Number of timestamp breaks after synchronization / Total number of data entries), where timestamp break is defined as: the interval between adjacent data timestamps exceeds the collection period; (6) Reward function The calculation method is as follows: .

[0053] It should be noted that the acquisition of data and collection of information in this application are legal, compliant, or [illegible]. Obtain the consent of the data subject.

[0054] The optional embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above embodiments. Within the scope of the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all fall within the protection scope of the embodiments of the present invention.

Claims

1. A method for real-time data synchronization of multi-target databases for power metering, characterized in that, Includes the following steps: Step 1: Use edge node devices to collect real-time data from power metering terminals, preprocess the real-time data from power metering terminals, and obtain the state matrix of energy consumption and resources, as well as the preprocessed data. The energy consumption and resource status matrix and the preprocessed data are uploaded to the cloud server together. The energy consumption and resource status matrix is ​​a two-dimensional data containing energy consumption and bandwidth generated by the power consumption sensor built into the edge node device to collect CPU power consumption and the bandwidth utilization rate collected by the network monitoring module. Step 2: After receiving the preprocessed data, the cloud server automatically detects the target library type and extracts the storage format and field constraint features; it uses a dynamic format conversion model based on abstract syntax tree to adapt the format of the preprocessed data, and uses the hierarchical analysis method to determine the target library adaptation priority, forming an adapted data and adaptation priority list. Step 3: Construct a five-dimensional objective function based on the energy consumption and resource state matrix and the adaptation priority list, and use the DDPG reinforcement learning algorithm to generate the optimal scheduling scheme; wherein, the five-dimensional objective function specifically includes the following: (1) Business priority objectives Weights are assigned based on billing and scheduling services. Synchronization timeliness is judged by ≤1s, with 1 being compliant and 0 being non-compliant. in, In the formula, i=1 represents billing services, i=2 represents scheduling services, i=3 represents operation and maintenance services, i=4 represents backup services, and W i As a business weight, For synchronization and timeliness; (2) Cross-database adaptation success rate target Calculated based on the adaptation results of the adapted data; in, =Amount of data successfully written to the target database / Total amount of synchronized data; (3) System energy consumption target It is calculated by summing the power-time products of the edge node devices and the cloud server; Among them, the power parameter is the monitoring data of edge node devices and cloud servers, the goal is to minimize total energy consumption, the power is the average power over 1 minute in W, and the time granularity is the duration of the synchronization task in h. (4) Network load balancing objectives The calculation method is as follows: =1-[(Maximum bandwidth utilization - Minimum bandwidth utilization) / Average bandwidth utilization]; Among them, the bandwidth utilization rate data is collected by edge node devices, with a target value ≥0.85; (5) Data time sequence consistency target The core principle is to ensure that the timestamps of synchronized data are unbroken. in, =1-(Number of timestamp breaks after synchronization / Total number of data entries), where timestamp break is defined as: the interval between adjacent data timestamps exceeds the collection period; (6) Reward function The calculation method is as follows: , During calculation, min-max normalization is used to map the data to the [0,1] interval; Step 4: Based on the optimal scheduling scheme, first predict the future 5-minute trend of the adapted data using an LSTM model. The voltage and current time series data sampled at 1Hz over the past hour are used as the training set. The Adam optimizer is used for training with a training rate of 0.

001. Only the data with predicted changes are processed using the sliding window difference algorithm. The sliding window size is set to 100 data points. The difference between the current data and the previous data in the window is calculated. Only the difference is transmitted. A lightweight compression algorithm and dynamic transmission path are selected based on the target library type to complete the data transmission. The LZ77 algorithm is used for time series databases, the Snappy algorithm is used for non-relational databases, and the Gzip algorithm is used for relational databases. Step 5: The transmitted data is segmented and hashed in real time using the SHA-256 hash algorithm. Each 1KB of data transmitted triggers an SHA-256 hash calculation, which is compared with the hash value pre-stored on the edge node device in real time through peer-to-peer comparison. If an inconsistency is found, the transmission is immediately interrupted and retransmission is initiated. After synchronization is completed, the verified data hash value, edge node device ID, and timestamp information are encapsulated into a transaction and sent to the power industry consortium chain node. After verification by the PBFT consensus algorithm, the transaction is written into a block. The block generation time is ≤3s.

2. The method for real-time data synchronization of multi-target databases for power metering according to claim 1, characterized in that, Step 1 specifically includes the following: Step 1.1: Use edge node devices to collect real-time data from power metering terminals; Step 1.2: Remove extreme outliers using the Isolation Forest algorithm, which incorporates temporal characteristics; Step 1.3: After removing extreme outliers, the valid data is distinguished into steady-state data and burst data. The steady-state data is aggregated by minute-level average, and the voltage fluctuation of the steady-state data is ≤0.5%. The burst data is retained as the original data to obtain the preprocessed data. Simultaneously monitor node energy consumption and network resource status to generate an energy consumption and resource status matrix; Step 1.4: Upload the preprocessed data and the energy consumption and resource status matrix to the cloud server.

3. The method for real-time data synchronization of multi-target databases for power metering according to claim 1, characterized in that, The target library types include MySQL, InfluxDB, MongoDB, and Oracle.

4. The method for real-time data synchronization of multi-target databases for power metering according to claim 1, characterized in that, In step 2, the analytic hierarchy process (AHP) is used to determine the priority of target library adaptation, as follows: A three-tiered indicator system is constructed, which includes three indicators: business importance (weight 0.6), storage capacity (weight 0.2), and write performance (weight 0.2). Among them, the business importance indicator is calculated by weighting the billing system (weight 0.4), scheduling system (weight 0.3), operation and maintenance system (weight 0.2), and backup database (weight 0.1). A priority list for adaptation is generated by combining the scores of various indicators.

5. A real-time data synchronization system for a multi-objective database for power metering, characterized in that, This includes edge node devices and cloud servers; the edge node devices and cloud servers are used to perform the following steps: Step 1: Use edge node devices to collect real-time data from power metering terminals, preprocess the real-time data from power metering terminals, and obtain the state matrix of energy consumption and resources, as well as the preprocessed data. The energy consumption and resource status matrix and the preprocessed data are uploaded to the cloud server together. The energy consumption and resource status matrix is ​​a two-dimensional data containing energy consumption and bandwidth generated by the power consumption sensor built into the edge node device to collect CPU power consumption and the bandwidth utilization rate collected by the network monitoring module. Step 2: After receiving the preprocessed data, the cloud server automatically detects the target library type and extracts the storage format and field constraint features; it uses a dynamic format conversion model based on abstract syntax tree to adapt the format of the preprocessed data, and uses the hierarchical analysis method to determine the target library adaptation priority, forming an adapted data and adaptation priority list. Step 3: Construct a five-dimensional objective function based on the energy consumption and resource state matrix and the adaptation priority list, and use the DDPG reinforcement learning algorithm to generate the optimal scheduling scheme; wherein, the five-dimensional objective function specifically includes the following: (1) Business priority objectives Weights are assigned based on billing and scheduling services. Synchronization timeliness is judged by ≤1s, with 1 being compliant and 0 being non-compliant. in, In the formula, i=1 represents billing services, i=2 represents scheduling services, i=3 represents operation and maintenance services, i=4 represents backup services, and W i As a business weight, For synchronization and timeliness; (2) Cross-database adaptation success rate target Calculated based on the adaptation results of the adapted data; in, =Amount of data successfully written to the target database / Total amount of synchronized data; (3) System energy consumption target It is calculated by summing the power-time products of the edge node devices and the cloud server; Among them, the power parameter is the monitoring data of edge node devices and cloud servers, the goal is to minimize total energy consumption, the power is the average power over 1 minute in W, and the time granularity is the duration of the synchronization task in h. (4) Network load balancing objectives The calculation method is as follows: =1-[(Maximum bandwidth utilization - Minimum bandwidth utilization) / Average bandwidth utilization]; Among them, the bandwidth utilization rate data is collected by edge node devices, with a target value ≥0.85; (5) Data time sequence consistency target The core principle is to ensure that the timestamps of synchronized data are unbroken. in, =1-(Number of timestamp breaks after synchronization / Total number of data entries), where timestamp break is defined as: the interval between adjacent data timestamps exceeds the collection period; (6) Reward function The calculation method is as follows: , During calculation, min-max normalization is used to map the data to the [0,1] interval; Step 4: According to the optimal scheduling scheme, first predict the future 5-minute change trend of the adapted data using the LSTM model. Only the data that is predicted to change is processed using the sliding window difference algorithm. Calculate the difference between the current data and the previous data in the window. Only transmit the difference part. Combine the target library type to select a lightweight compression algorithm and a dynamic transmission path to complete the data transmission. Step 5: The transmitted data is segmented and hashed in real time using the SHA-256 hash algorithm. Each 1KB of data transmitted triggers an SHA-256 hash calculation, which is compared with the hash value pre-stored on the edge node device in real time through peer-to-peer comparison. If an inconsistency is found, the transmission is immediately interrupted and retransmission is initiated. After synchronization is completed, the verified data hash value, edge node device ID, and timestamp information are encapsulated into a transaction and sent to the power industry consortium chain node. After verification by the PBFT consensus algorithm, the transaction is written into the block.

Images