Adaptive data consistency maintenance method for electric power information system based on incremental updating mechanism

Abstract

The invention provides a self-adaptive data consistency maintenance method for an electric power information system based on incremental updating, and relates to the technical field of big data processing, and the method comprises an initialization step, a detection step and a feedback step. In the initialization step, a network including regional power data nodes and a distributed account book is constructed, and full power data is stored; in the detection step, power data changes are confirmed, key attributes are used for marking data, automatic conflict detection is carried out through a distributed consensus algorithm, solutions for coping with conflicts are dynamically selected, and then an incremental updating mechanism is adopted for synchronously changing the data; in the feedback step, a monitoring system with a closed-loop feedback mechanism is established, the data consistency state is automatically tracked when the electric power data is changed, and abnormity is recognized. Compared with a traditional data synchronization method which needs to synchronize full data, the method provided by the invention only synchronizes the changed data fragments through an incremental updating mechanism, thereby ensuring the consistency of the data, remarkably improving the data processing efficiency and saving the network bandwidth.

Description

Technical Field

[0001] This invention relates to the field of power system automation and smart grid big data processing, and in particular to an adaptive data consistency maintenance method for power information systems based on an incremental update mechanism. This method is applicable to cross-regional power data synchronization management scenarios and solves the data consistency problem under massive power equipment monitoring data. Background Technology

[0002] With the rapid advancement of smart grid construction, the amount of data that modern power information systems need to process is growing exponentially. Traditional full data synchronization methods can no longer meet the data storage and processing requirements of power information systems. In this big data environment, the problem of power data consistency is becoming increasingly prominent: traditional two-phase commit protocols require the full transmission of data, resulting in huge bandwidth and storage resource consumption in power communication networks; fixed synchronization strategies cannot adapt to the dynamic environment of grid load fluctuations and communication delay changes, requiring nodes to frequently write full power data, leading to a surge in disk I / O and CPU resource consumption and insufficient dynamic adaptability; when a node fails, full data reconstruction takes a long time, has low recovery efficiency, and affects the performance of power information systems.

[0003] In critical business scenarios of power systems, in order to ensure that users obtain consistent information on different devices, traditional data consistency maintenance methods employ strong consistency models and eventual consistency models:

[0004] In a strong consistency model, all data updates are atomic at the application layer, and any read will return the latest write. While this approach effectively avoids data inconsistencies, it is costly to implement in widely distributed power monitoring systems, especially in emergency power grid control scenarios. When system load increases, system response time lengthens, protection actions are not executed in a timely manner, triggering cascading failures. The numerous locking mechanisms and transaction management also increase the system burden, thereby affecting business processing speed and leading to a decline in user experience.

[0005] In the eventual consistency model, short-term data inconsistencies are allowed, but eventually all data will reach a consistent state. This approach is relatively flexible and lenient, and has good performance advantages for scenarios with high-concurrency data writing, such as widely distributed power information systems. However, in application scenarios with high data consistency requirements, such as power grid orchestration and protection control, short-term data inconsistencies may lead to misunderstandings among users, resulting in economic losses and serious consequences.

[0006] Considering the aforementioned limitations, existing technologies have initially attempted to use incremental synchronization mechanisms to perform difference comparisons in certain systems to maintain data consistency. Currently, this method is only applicable to the file level and lacks universality in distributed scenarios. Therefore, this invention addresses the inherent shortcomings of full synchronization by transmitting only the changed data portion to reduce network and storage overhead, adjusting the synchronization strategy based on real-time load and network status, and combining anomaly detection and automated repair to shorten fault recovery time. Summary of the Invention

[0007] To address the shortcomings of the aforementioned technologies, the purpose of this invention is to solve the performance bottleneck problem caused by full updates and inflexible consistency strategies in the data consistency maintenance process of power information systems.

[0008] To achieve the above objectives, the technical solution adopted by this invention is as follows: an adaptive data consistency maintenance method for a power information system based on an incremental update mechanism, comprising an initialization step, a detection step, and a feedback step. The initialization step constructs a network including regional power data nodes and a distributed ledger; the detection step confirms data changes and marks the data, performs automatic conflict detection, dynamically selects a solution, and synchronously changes the data; the feedback step establishes a real-time monitoring system to automatically track the power data consistency status of each regional node and identify anomalies.

[0009] Optionally, the initialization step specifically includes:

[0010] The design concept combines a chain structure and a Merkle tree to link the data blocks together. Each block contains the root hash value of a Merkle tree and several transactions to ensure data integrity and consistency.

[0011] To achieve effective connectivity and verification between power data blocks, newly added blocks are linked to the previous blocks in the order of the chain, thus forming a continuous data chain.

[0012] Each power data block not only stores the data, but also includes its metadata, including the data's variability and related hash values, to ensure data security and consistency;

[0013] In the process of forming a data chain, distributed computing and network methods are used to ensure the consistent state of all data nodes;

[0014] Building on this, the process is combined with the multi-node architecture of NoSQL databases to meet the needs of data storage and efficient retrieval, thereby enhancing the system's scalability.

[0015] Furthermore, upon initial startup, a primary regional node is randomly selected as the leader, and the entire power data is divided into blocks, with the Rabin fingerprint of each block calculated:

[0016]

[0017] In the formula, R(b) is the fingerprint value, representing the unique identifier of the data block, b is the data block, n is the length of the power data block, i.e., the number of elements that make up the data block, and b i This represents the i-th byte or character in data block b. p is a base greater than all possible values ​​of elements in the data block. It is usually chosen as a prime value to reduce the possibility of collisions. Generally, p is chosen as 31 or 257. M is the modulus.

[0018] Choose the appropriate communication protocol based on the needs, performance, and architecture of the power communication network. HTTP / 2 is used for power orchestration applications and traditional RESTful APIs, gRPC is used for microservice architectures and real-time monitoring applications, WebSocket is used for high-frequency power data acquisition to support data transmission and real-time communication, and message queues Kafka or RabbitMQ are used to handle message passing between nodes, ensuring power data consistency and system availability. Configure database and storage systems so that each node can access its full copy of data and ensure stable connections between nodes.

[0019] In this process, power data blocks are distributed to other nodes via gRPC, and each node stores the data after verifying the Merkle root hash.

[0020] Optionally, the detection step specifically includes:

[0021] When a node needs to perform a power data update, the leader, based on orchestration instructions or system behavior, first generates an incremental update proposal, including the power data object to be updated and its state identifier. By comparing the fingerprint sets of the old and new power data, it identifies changed and unchanged blocks. For unchanged blocks, it generates a reference instruction, and for changed blocks, it generates an incremental Delta representing the difference. Finally, it selects a differentiated incremental encoding strategy based on the applicable scope.

[0022] The data consists of power equipment status data and binary monitoring data. Using the VCDIFF algorithm, the differences between the old and new data are calculated. Then, a sliding window matching technique is used to divide the old and new data into fixed-size windows. The new data is then searched for portions that match the old data window. Reference instructions are generated for the matching portions, and addition instructions are generated for the non-matching portions. These instruction sequences are encoded into incremental data.

[0023] Window matching in the algorithm:

[0024] Delta = Base ⊕ Target

[0025] In the formula, Delta represents the difference data, Base is the original baseline power data block, Target is the target power data block, representing the desired final data state, and ⊕ represents the XOR operation.

[0026] If the data consists of structured data such as power grid operation databases and power documents, then the operation log method is used to load the initial basic power data state D0 from the storage system or database. The operation log is used to record every operation on the data, including insertion, deletion, and update, thereby generating operation tuples O. i :O i =(t i ,type i ,pos i ,content i ),

[0027] Among them, t i For timestamps, type i For operation type, pos i For the operation location, content i This refers to the operation content; when performing power data operations, the data increment ΔD generated based on the operation log will be applied to the current power data status:

[0028] D new =D0 + ΔD, using a distributed consensus protocol to update the data state D. new Perform data consistency checks; after acquiring incremental data, use the LZ77 compression algorithm on the leader node to find the longest matching string of processed power data in a sliding window using duplicate data, thereby performing compression. The matching function is:

[0029]

[0030] Match represents the starting position and length of the best match found. This indicates that the combination of i and j is being searched to maximize the length function. The len function represents the length of the substring. S[i:j] represents a substring from position i to j in the power data stream or input string S. This substring is the part currently being examined. LZ77 will search for matches between the substring and the data already in the sliding window. T[k:l] represents a segment of data in the sliding window, where k and l represent the start and end positions of the segment, respectively.

[0031] The node used for dividing the full power data into blocks encapsulates the incremental update information into a proposal, and then sends the proposal to all follower nodes via the efficient multicast protocol gRPC.

[0032] During the proposal sending process, the system continuously monitors the status of each node through the fault detection and recovery mechanism of the distributed consensus algorithm. If a node is found to be unresponsive, it is determined to be a faulty node.

[0033] If the currently failed node is the leader node, the system will conduct a new leader election based on the distributed consensus protocol. The majority of nodes will confirm the integrity of the candidate logs, thereby electing a new node as the leader to maintain the continuity of the system. The failed node can rebuild its state by replaying the operation logs or resume transmission from a breakpoint, synchronizing only the missing incremental fragments. This ensures that even if some nodes fail, the system can continue to run without losing power data consistency.

[0034] After receiving a proposal, a follower node determines the validity of the protocol for itself according to the following logic:

[0035] Check if the version numbers of the proposals are consecutive and verify the completeness of the incremental changes;

[0036] Incremental updates are performed locally on a temporary application to simulate the updated state. If the pre-execution is successful and meets the business rules, then the application is approved; otherwise, it is rejected.

[0037] The voting results are returned to the leader as a message, and the results are recorded in the leader's own state.

[0038] Leaders make decisions after collecting votes from their followers:

[0039] If more than half of the nodes vote in favor, the proposal is confirmed to have passed.

[0040] If a majority consensus is not reached, different solutions will be chosen depending on the specific scenario:

[0041] If it is a network partition scenario, then a leader re-election will be triggered;

[0042] If there is a data conflict, the incremental data will be rolled back and the conflict resolution process will be triggered.

[0043] Once the proposal is approved, the leader broadcasts the approval message to all nodes.

[0044] After receiving confirmation, other nodes, under the guidance of the leader, only modify the change fields specified in the proposal, without needing to overwrite the entire proposal, and update the local version number and persist the logs to ensure traceability;

[0045] If a follower does not receive a confirmation message, a synchronization recovery is triggered via heartbeat detection, pulling the missing incremental updates from the leader or other replicas.

[0046] Before a node makes a data change, a set of key attributes is assigned to each data block to identify the uniqueness and state of the data. When multiple nodes operate on the same data, decisions can be made based on the weights of the key attributes, and the synchronization and consistency between nodes are guaranteed through a distributed consensus algorithm.

[0047] When dynamically evaluating the consistency state during data synchronization, the consistency state equation is used:

[0048] D t =aD t-1 +(1-a)D s ,

[0049] In the formula, D t D represents the power data status at the current point in time. t-1 D represents the power data status at the previous time point. s This is synchronous power data from other nodes. 'a' is a weighting factor, which defaults to 0.5 and is used to balance the relative importance of local and synchronous data.

[0050] According to the adjustment rules:

[0051] Increasing α when network latency is high indicates reliance on local data; decreasing α when latency is low indicates priority for synchronizing data.

[0052] When |D t -D s When |>∈, the conflict resolution process is triggered.

[0053] Where ∈ represents the allowable error threshold. Considering the needs of power business, data characteristics, and system tolerance, ∈ can be designed as a dynamic value.

[0054] If based on load:

[0055]

[0056] In the formula ∈ 基线 The baseline allowable error threshold is set according to the criticality level of power data in this invention, such as critical power parameters ∈ 基线 =0, non-critical monitoring data ∈ 基线 >0;

[0057] If based on data type:

[0058] Key power parameters are set to ∈=0, and non-key monitoring data are set to ∈=+∞.

[0059] Furthermore, regarding the consistency overhead problem, appropriate solutions are selected based on different scenarios. In the distributed system of this invention, the tree-structured synchronization method requires multiple layers of forwarding, leading to accumulated latency. The uniform synchronization method may require full synchronization or complex topology maintenance. While the latency-biased synchronization method prioritizes low-latency paths, it does not optimize data volume, resulting in performance degradation during network congestion. In contrast, the key attribute-based optimization method, namely OCKP, has the lowest consistency overhead due to incremental updates and key attribute filtering, making it particularly suitable for large-scale distributed systems. Through the explicit definition of the key attribute set and semantic update description, it achieves accurate identification and processing of key updates.

[0060] Each type of resource Res predefines its key attribute set KS(Res), which contains all the core fields that affect business logic. The update operation Updatei,j(DE) submitted by the user or system is described as a tuple.<Ii,j,k(DE),Ui,j,k(DE)> The set of data objects, where DE is the data object identifier, i is the update publishing node ID, j is the update sequence number, and it is assumed that the data objects on each node are allocated in units of data objects. Ii,j,k(DE) is a description of the updated data information of data object DE, Ui,j,k(DE) is the corresponding information after the data object is modified, and k is the number of the modified item. If Updatei,j(DE) satisfies:

[0061] Then it is considered that the update operation Updatei,j(DE) is a critical update to the data object DE.

[0062] Furthermore, heat calculations are used to dynamically adjust power data migration and load balancing, thereby ensuring the validity of each data block:

[0063] Heat value H i Starting with the access frequency of power data blocks, for each data block D... i Count the number of visits within the time window T and calculate the popularity value H. i :

[0064]

[0065] Based on the higher weight of recent visits and the difference in weight of read and write operations, the popularity value is dynamically weighted to refine the data priority. This is used to assist in the selection of migration order and load distribution when generating solutions in subsequent detection steps, thereby improving adaptability.

[0066] H i ′ =H i ×α 时间 ×β 类型 ,

[0067] Among them, H i ′ For the weighted heat value, α 时间 β is the time weighting coefficient. 类型 This is the type weight coefficient.

[0068] The relationship between popularity and load changes over time depending on different network needs. It can be dynamically adjusted using thresholds based on system status and access patterns.

[0069] Specifically, the formula for adjusting the threshold is as follows:

[0070] If based on a time window, the access frequency is counted in different time windows to adjust the threshold:

[0071] Threshold=μ+k·σ,

[0072] In the formula, μ is the average access frequency, k is the adjustment coefficient, which is usually taken as 1 to 3, and σ is the standard deviation of the access frequency. If based on the load, when the system load is high, the threshold is reduced, that is, the adjustment coefficient k is reduced to migrate more hot power data to distribute the pressure. Conversely, the threshold is increased, that is, the adjustment coefficient k is increased to reduce migration overhead.

[0073] Among them, if the heat value H i If the current data block D is greater than or equal to the threshold, then... i For hot data, a strong consistency protocol is used, and it is stored on high-performance media;

[0074] If the heat value H i If the value is less than the threshold, then the current data block D... i For cold data, an eventual consistency protocol is used, and the data is stored on a low-cost medium.

[0075] Furthermore, regarding data block D i Overall popularity value H i and migration cost C i Calculate its migration priority P i :

[0076] P i =H i ·C i ,

[0077] Sort by priority from highest to lowest, and migrate the highest priority power data blocks first.

[0078] Furthermore, the load distribution is dynamically adjusted based on the node load. For each node N... j Its load weight W j It is inversely proportional to its current load:

[0079]

[0080] In the formula Load(N) j ) represents node N j The current load status can be evaluated by combining indicators such as CPU utilization, memory usage, and network bandwidth.

[0081] Based on the above heat value and load weight, data block D i The access request is assigned to node N j The probability P ij for:

[0082] In the formula, ∑ k H i ·W k Calculate the sum of the heat weights of all nodes for standardization.

[0083] The dynamic adjustment of data migration and load balancing strategies by the aforementioned heat ranking algorithm involves the following dynamic adjustments and exception handling:

[0084] Periodically recalculate μ and σ, and dynamically update the threshold, such as every 5 minutes;

[0085] If a node fails, the data blocks that need to be migrated are quickly located based on their popularity value, and hot data is restored first. Optionally, the feedback step specifically includes:

[0086] Monitoring agents are deployed at various regional nodes of the power information system. These agents are configured with appropriate sampling rates and use Prometheus to collect data on CPU utilization, memory usage, disk I / O, and network bandwidth of each node. The Elastic Stack is used to collect and analyze power system logs and event logs.

[0087] For key power indicators, the sampling rate can be set to once per second or once every two seconds. For indicators that are updated less frequently, it can be set to once per minute. For log data, it can be set to once per minute to once per hour.

[0088] The monitoring agent periodically collects local status data and pushes information to the central monitoring system.

[0089] The central monitoring system analyzes the collected data using the 3-Sigma rule in anomaly detection algorithms. Based on the normal distribution assumption and a sliding window, it identifies data exceeding the range of μ±3σ as anomalous data; the anomaly condition is: x<μ-3σ or x>μ+3σ.

[0090] The mean μ is the center of the distribution, and the standard deviation σ measures the degree of dispersion of the data.

[0091] When the indicators exceed the preset range, the system can automatically identify the abnormal state and start the recovery process, such as re-electing a leader or re-synchronizing power data.

[0092] Compared to existing technologies, the advantages of this invention are as follows: 1. By using incremental updates and Rabin fingerprinting technology, changed blocks are quickly identified, avoiding invalid transmissions and transmitting only the differences, significantly reducing data transmission volume and network bandwidth consumption by about 70%, making it particularly suitable for medium- to high-frequency update scenarios; 2. By dynamically adjusting strategies to dynamically calculate the heat value of data blocks, while ensuring the consistency strength of critical data is maintained in the range of 0.4 to 0.7, the synchronization overhead of non-critical data is reduced, improving the overall system throughput, and reducing node load differences to 20% to 50%, avoiding performance bottlenecks under high load; 3. Core fields affecting business logic are clearly defined, and strong consistency processes are triggered only for critical conflicts. By identifying the marking of key attributes and automating resolution, manual intervention is reduced, increasing the conflict detection accuracy from 70% to about 87%, and improving system response speed by about 80%. In abnormal situations, real-time monitoring and 3-Sigma rules dynamically identify faulty nodes and abnormal data. Through incremental recovery and rapid leader election, second-level fault switching is achieved, and the recovery time is shortened by about 98% compared to the time for full data recovery using traditional methods, ensuring high system availability. By using dynamic load balancing and incremental synchronization, single-point overload is avoided, data migration costs are reduced by approximately 85%, and the system can flexibly adapt to changes in the number of replicas to achieve dynamic expansion. Attached Figure Description

[0093] Figure 1 This is a schematic diagram illustrating the solution process of an adaptive data consistency maintenance method for power information systems based on an incremental update mechanism.

[0094] Figure 2 For the overall architecture of power information system management;

[0095] Figure 3 A flowchart for data orchestration;

[0096] Figure 4 Comparison of consistency overhead when the number of replicas is fixed;

[0097] Figure 5 This is a schematic diagram of the abnormal recovery process. Detailed Implementation

[0098] The present invention will be further described below with reference to an embodiment of a power information system.

[0099] Example: Figure 1As shown, in the power information system, a network architecture comprising multiple power data nodes and a distributed ledger is first constructed. The system adopts a three-tier deployment model: the provincial orchestration center acts as the master node, municipal power supply companies as regional nodes, and substations as edge nodes, forming a power data synchronization system covering the entire network. Specifically, the following node types are deployed in the system: orchestration service nodes, deployed at the provincial dispatch center, responsible for handling read and write requests for power data across the entire network; each node stores a complete copy of the power data to ensure data redundancy and high availability; regional service nodes, deployed at municipal power supply companies, handle regional power data acquisition and processing, and interact with the orchestration center node; monitoring nodes, collecting grid operation indicators, are deployed at substations of various voltage levels.

[0100] A distributed ledger is designed using a combination of a chain structure and a Merkle tree. Each block contains the root hash of the Merkle tree, the hash of the previous block, a timestamp of the electricity data, and a data source identifier. Hash pointers connect the blockchain to form a structure, enabling effective linking and verification between electricity data blocks. Grid state change records are stored as transaction data between blocks. Each time a block is updated, the new block is linked to the previous block according to the chain order, the Merkle tree hash is recalculated, and the new root hash is stored in the new block. Each power device corresponds to a data block containing key operating parameters and operation logs. During the blockchain formation process, a multi-node architecture with a NoSQL database is integrated to enhance the system's scalability.

[0101] Upon initial system startup, a master regional node is elected using a modified Raft protocol based on a distributed consensus algorithm. The election process comprehensively considers factors such as node geographical location, communication link quality, hardware configuration, and current load status. After election, the master node divides the entire network's power data into blocks according to the following rules: SCADA telemetry data is grouped by substation and equipment type, with one data block every 5 minutes; equipment ledger data is grouped by equipment type and region, with one data block for every 100 records; fault recording files are divided into fixed 2MB blocks. After block division, the Rabin fingerprint of each block is calculated.

[0102]

[0103] In the formula, R(b) is the fingerprint value, representing the unique identifier of the data block, b is the power data block, n is the length of the data block, i.e., the number of elements that make up the data block, and b i This represents the i-th byte or character in data block b. p is a base greater than all possible values ​​of elements in the data block. It is usually chosen as a prime value to reduce the possibility of collisions. Generally, p is chosen as 31 or 257. Here, 257 is chosen to reduce the probability of electrical data collisions. M is the modulus.

[0104] Based on the communication characteristics of the power system, the HTTP / 2 protocol is selected for orchestrating command transmission, the gRPC protocol for efficient data synchronization between nodes, and the Kafka protocol for message queuing to handle high-concurrency power data changes. Power data blocks are distributed to other nodes via gRPC, and each node stores the data after verifying the Merkle root hash.

[0105] Combination Figure 2 and Figure 3 When substation voltage fluctuates, edge nodes detect the voltage change and use the VCDIFF algorithm to directly calculate the difference between old and new data. Through a sliding window matching technique, the old and new data are divided into fixed-size windows. The algorithm searches for parts in the new data that match the old data window. Reference instructions are generated for matching parts, and addition instructions are generated for non-matching parts. The instruction sequence is encoded as incremental data. The window matching in the algorithm is as follows:

[0106] Delta = Base ⊕ Target

[0107] In the formula, Delta represents the difference data, Base is the original baseline power data block, Target is the target power data block, representing the final data state that is desired to be achieved, and ⊕ represents the XOR operation;

[0108] After obtaining the incremental data Delta, the LZ77 compression algorithm is used to find the longest matching string in a sliding window using duplicate data, thereby compressing the incremental update packet and generating a proposal. The matching function is:

[0109] Match represents the starting position and length of the best match found. This indicates the search for the combination of i and j that maximizes the length function. The len function represents the substring length. S[i:j] represents a substring from position i to j in the data stream or input string S. This substring is the part currently being examined. LZ77 will search for matches between the substring and the data already in the sliding window. T[k:l] represents a segment of data in the sliding window, where k and l represent the start and end positions of the segment, respectively.

[0110] The master node sends the proposal to all follower nodes via the efficient multicast protocol gRPC.

[0111] When a proposal is transmitted in the network, the system continuously monitors the status of each node through the fault detection and recovery mechanism of the distributed consensus algorithm. If a node is found to be unresponsive, it is determined to be a faulty node. If the current leader node fails, the system conducts a new leader election based on the distributed consensus protocol. The majority of nodes confirm the integrity of the candidate logs, thereby electing a new node as the leader to maintain the continuity of the system. Failed nodes can reconstruct their state by replaying the operation logs or resume transmission from a breakpoint, synchronizing only the missing incremental segments. This ensures that even if some nodes fail, the system can continue to run without losing data consistency.

[0112] After receiving a proposal, a follower node checks if the proposal's version number is consecutive, verifies whether the voltage change complies with power grid safety rules, simulates the voltage adjustment, and if the voltage is within acceptable limits after the pre-execution, it votes in favor. The follower node returns the voting result to the master node and records it in its local log.

[0113] The master node tallies the voting results. If it collects more than half of the nodes' votes in favor, the proposal is confirmed as passed, and the confirmation information is broadcast to the entire network.

[0114] After receiving confirmation from the master node, the follower node makes data changes, only modifying the fields specified in the proposal, without overwriting the entire data, and updates the local version number and persists the logs to ensure traceability.

[0115] If a follower does not receive a confirmation message, a synchronization recovery is triggered via heartbeat detection, pulling the missing incremental updates from the leader or other replicas.

[0116] If a majority consensus is not reached, different solutions are selected based on the specific scenario. In the case of a network partition scenario, a leader re-election is triggered. In the case of data conflict, incremental changes are rolled back and a conflict resolution process is triggered until a new consensus is reached.

[0117] When dynamically evaluating the consistency state during data synchronization, the consistency state equation is used: D t =aD t-1 +(1-a)D s ,

[0118] In the formula, D t D represents the data state at the current point in time. t-1 D represents the local data state at the previous point in time. s This is the synchronized data from other nodes. 'a' is a weighting factor that is dynamically adjusted based on the power communication quality. The default value is 0.5, used to balance the importance of local and remote data.

[0119] According to the adjustment rules, α is increased when network latency is high, indicating reliance on local data; α is decreased when latency is low, indicating priority for data synchronization.

[0120] When |D t -D s When |>∈, the conflict resolution process is triggered.

[0121] Where ∈ is the allowable error threshold. According to the design concept of adaptive adjustment of system state, ∈ can be designed as a dynamic value:

[0122] If adjustments are based on load:

[0123]

[0124] In the formula ∈ 基线 The baseline allowable error threshold is set according to the data criticality level in this invention, such as key fields ∈ 基线 =0, non-key fields ∈ 基线 >0;

[0125] If the adjustment is based on the data type:

[0126] Set ∈=0 for critical fields and ∈=+∞ for non-critical fields.

[0127] Furthermore, through the explicit definition and semantic update description of key attribute sets, accurate identification and processing of key updates are achieved. Regarding consistency overhead issues, appropriate solutions are selected based on different scenarios. In the distributed system of this invention, the tree-structured synchronization method requires multi-layer forwarding, leading to latency accumulation; the uniform synchronization method may require full synchronization or complex topology maintenance; and while the latency-biased synchronization method prioritizes low-latency paths, it does not optimize data volume, resulting in performance degradation during network congestion. Figure 4 In comparison, the key attribute-based optimization method, OCKP, has the lowest consistency overhead due to incremental updates and key attribute filtering, making it particularly suitable for large-scale distributed systems.

[0128] Each type of resource Res predefines its key attribute set KS(Res), which contains all the core fields that affect business logic. The update operation Updatei,j(DE) submitted by the user or system is described as a tuple.<Ii,j,k(DE),Ui,j,k(DE)> The set of data objects, where DE is the data object identifier, i is the update publishing node ID, j is the update sequence number, and it is assumed that the data objects on each node are allocated in units of data objects. Ii,j,k(DE) is a description of the updated data information of data object DE, Ui,j,k(DE) is the corresponding information after the data object is modified, and k is the number of the modified item. If Updatei,j(DE) satisfies:

[0129]

[0130] Then it is considered that the update operation Updatei,j(DE) is a critical update to the data object DE.

[0131] Furthermore, data migration and load balancing strategies are dynamically adjusted using heat calculations to ensure the validity of each data block:

[0132] Heat value H i Starting with the access frequency, for each data block D i Count the number of visits within the time window T and calculate the popularity value H. i :

[0133]

[0134] Based on the higher weight of recent visits and the difference in weight of read and write operations, the popularity value is dynamically weighted to refine the data priority. This is used to assist in the selection of migration order and load distribution when generating adjustment plans in subsequent detection steps.

[0135] H i ′ =H i ×α 时间 ×β 类型 ,

[0136] Among them, H i ′ For the weighted heat value, α 时间 β is the time weighting coefficient. 类型 This is the type weight coefficient.

[0137] The relationship between heat value and load changes over time to meet different network needs. It can be dynamically adjusted based on system status and access patterns using thresholds.

[0138] Specifically, the formula for adjusting the threshold is as follows:

[0139] If based on a time window, the access frequency is counted in different time windows to adjust the threshold:

[0140] Threshold = μ + k·σ, where μ is the average access frequency, k is the adjustment coefficient (usually between 1 and 3), and σ is the standard deviation of the access frequency. If based on load, when the system load is high, the threshold is lowered (i.e., the adjustment coefficient k is lowered) to migrate more hotspot power data to distribute the load; conversely, the threshold is raised (i.e., the adjustment coefficient k is raised) to reduce migration overhead.

[0141] Among them, if the heat value H i If the current data block D is greater than or equal to the threshold, then... iFor hot data, a strong consistency protocol is used, and it is stored on high-performance media;

[0142] If the heat value H i If the value is less than the threshold, then the current data block D... i For cold data, an eventual consistency protocol is used, and the data is stored on a low-cost medium.

[0143] Furthermore, regarding data block D i Overall popularity value H i and migration cost C i Calculate its migration priority P i :P i =H i ·C i ,

[0144] Sort by priority from high to low, and migrate high-priority data blocks first.

[0145] Furthermore, the load distribution is dynamically adjusted based on the node load. For each node N... j Its load weight W j It is inversely proportional to its current load:

[0146]

[0147] In the formula Load(N) j ) represents node N j The current load status can be assessed by combining CPU utilization, memory usage, and network bandwidth.

[0148] Based on the above heat value and load weight, data block D i The access request is assigned to node N j The probability P ij for:

[0149] In the formula, ∑ k H i ·W k Calculate the sum of the heat weights of all nodes for standardization.

[0150] The dynamic adjustment of data migration and load balancing strategies by the aforementioned heat ranking algorithm involves the following dynamic adjustments and exception handling:

[0151] Periodically recalculate μ and σ, and dynamically update the threshold, such as every 5 minutes;

[0152] If a node fails, the data blocks that need to be migrated can be quickly located based on the popularity value, and hot data should be restored first.

[0153] Deploy a monitoring agent on each node, set an appropriate sampling rate, and use Prometheus to collect metrics such as CPU utilization, memory usage, disk I / O, and network bandwidth on each node. Use the Elastic Stack to collect and process logs. For key metrics, the sampling rate can be set to once per second or once every two seconds. For metrics that are updated less frequently, it can be set to once per minute. For log data, it can be set to once per minute to once per hour.

[0154] The monitoring agent periodically pushes the collected local state data to the central monitoring system. The central monitoring system uses the power-specific 3-Sigma rule in the anomaly detection algorithm to analyze the collected data. Based on the normal distribution assumption and sliding window, it identifies data exceeding the range of μ±3σ as anomaly data.

[0155] Abnormal conditions: x < μ - 3σ or x > μ + 3σ

[0156] The mean μ is the center of the distribution, and the standard deviation σ measures the degree of data dispersion.

[0157] When the metrics exceed the preset range, the system can automatically identify the abnormal state and start the recovery process to resynchronize the data, or trigger Raft election to attempt recovery when the leader node fails.

[0158] Combination Figure 5 Prometheus collects CPU utilization data per second. During this period, the CPU utilization of a substation node suddenly spiked to 90%. Under the 3-Sigma rule, the historical mean μ is 40% and the standard deviation σ is 15%. Based on the anomaly:

[0159] 90 > 40 + 3 × 15 = 85, triggering an alarm, isolating the faulty node, and the remaining nodes elect a new normal node as the master node within 5 seconds based on the Raft protocol; after the faulty node recovers, it pulls the latest power data from the new master node, only synchronizing the differences, and completing incremental data synchronization within 10 seconds without full replication.

[0160] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention. Equivalent transformations or substitutions made based on the above technical solutions all fall within the scope of protection of the claims of the present invention.

Claims

1. An adaptive data consistency maintenance method for power information systems based on an incremental update mechanism, characterized in that, The method includes the following steps: The initialization steps involve constructing a network architecture including data nodes and a distributed ledger within the power information system environment, and maintaining a full copy of the power data for each regional node. The detection process involves confirming changes in power data when data changes are triggered by power grid dispatch instructions, marking data through key attributes, using a distributed consensus algorithm for intelligent conflict detection, automatically identifying data conflicts and selecting solutions. The feedback process involves establishing a real-time monitoring system within a distributed system to continuously track the consistency of power data across nodes in each region, automatically identify abnormal data changes, and reselect a solution through a feedback mechanism.

2. The adaptive data consistency maintenance method for power information systems based on incremental update mechanism according to claim 1, characterized in that, The initialization steps construct a network including a distributed ledger, data nodes, and network communication protocols, where: A distributed ledger format is designed using a combination of chain structure and Merkle tree. Each change in electricity data is stored as a block transaction and linked to the previous block via hash pointers, forming a blockchain. This chain structure records blocks, and each block contains the root hash of a Merkle tree and several transactions. Each time a block is updated, the hash value of the Merkle tree is recalculated, and the new root hash is stored in the new block, realizing effective linking and verification between data blocks and ensuring data security and consistency. A distributed consensus algorithm ensures consensus on the ledger state among all data nodes in the network. Each data node retains a complete copy of the blockchain, ensuring the non-lossability and fault tolerance of electricity data. NoSQL databases are used to add more nodes, or key-value stores are used to simplify the data structure to adapt to high-concurrency read and write requests and provide horizontal scalability. During the initial startup, a primary regional node is randomly selected as the leader, and the entire power data is divided into blocks, with the Rabin fingerprint of each block calculated: In the formula, R(b) is the fingerprint value, representing the unique identifier of the data block, b is the power data block, n is the length of the data block, i.e., the number of elements that make up the data block, and b i This represents the i-th byte or character in the power data block b. p is a base greater than all possible values ​​of all elements in the data block. Prime values ​​are usually chosen to reduce collisions. p is either 31 or 257. M is the modulus.

3. The adaptive data consistency method for power information systems based on incremental update mechanism according to claim 2, characterized in that, In the detection process, the distributed consensus algorithm, combined with the incremental update mechanism, performs the following steps: When a node needs to perform a power data update, the leader, based on orchestration instructions or system behavior, first generates an incremental update proposal, typically including the data object to be updated and a state identifier. Then, it compares hash values ​​using the Rabin fingerprint algorithm, identifies changed blocks by comparing old and new power data, generates reference instructions for unchanged blocks, and generates an incremental Delta representing the difference for changed blocks. Finally, it selects an appropriate incremental encoding method based on the applicable scope. If applicable to power equipment status data and binary monitoring data, the VCDIFF algorithm is used to directly calculate the difference between the old and new data. A sliding window matching technique is employed to divide the old and new data into fixed-size windows. The algorithm searches for portions in the new data that match the old data window. Reference instructions are generated for the matching portions, and addition instructions are generated for the non-matching portions. The instruction sequence is encoded as incremental data. The window matching in the algorithm is as follows: Delta = Base ⊕ Target In the formula, Delta represents the difference data, Base is the original baseline power data block, Target is the target power data block, representing the final data state that is desired to be achieved, and ⊕ represents the XOR operation; If applicable to structured data such as power grid operation databases and power documents, the operation log method is used to load the initial basic power data state D0 from the storage system or database. The operation log records every operation on the data, including insertions, deletions, and updates. Each data operation is recorded as a tuple O. i : O i =(t i ,type i ,pos i ,content i ), Where: t i For timestamps, type i For operation type, pos i For the operation location, content i For the content of the operation; When performing power data operations, the data increment ΔD generated based on the operation log will be applied to the current power data state: D new =D0+ΔD, Updated power data status D new Use distributed consensus protocols to check data consistency.

4. The adaptive data consistency method for power information systems based on incremental update mechanism according to claim 3, characterized in that, After acquiring incremental data, the leader node uses the LZ77 compression algorithm to find the longest match in a sliding window that is identical to the content of the existing dictionary area, i.e., the processed power data, using duplicate data, thereby performing compression: Match represents the starting position and length of the best match found. Let i and j represent the combination of i and j that maximizes the length function. The len function represents the length of the substring. S[i:j] represents a substring from position i to j in the power data stream or input string S. This substring is the part currently being examined. LZ77 will search for matches between the substring and the data already in the sliding window. T[k:l] represents a segment of power data in the sliding window, where k and l represent the start and end positions of the segment, respectively.

5. The adaptive data consistency method for power information systems based on incremental update mechanism according to claim 4, characterized in that, The leader node broadcasts the compressed incremental data to the follower repository nodes via gRPC for verification and storage. During the proposal submission process, the system continuously monitors the status of each node through a fault detection and recovery mechanism based on a distributed consensus algorithm. If a node is detected as unresponsive, it is determined to be a faulty node. If the currently failed node is the leader node, the system will conduct a new leader election based on the distributed consensus protocol. A majority of nodes will confirm the integrity of the candidate's logs, thereby electing a new leader to maintain system continuity. Failed nodes can reconstruct their state by replaying operation logs or by resuming interrupted data transmissions, synchronizing only the missing incremental segments. This ensures that even if some nodes fail, the system can continue to operate without losing data consistency. Upon receiving the proposal, the follower will determine the validity of the protocol for itself based on its own state, record the voting result in its own state, and then send the voting result back to the leader in the form of a message. To prevent nodes from failing to synchronize in a timely manner due to network latency, downtime, or malfunctions, the leader employs a unified voting mechanism to collect votes from followers. Based on the principle of "more than half," the leader confirms the proposal's acceptance when more than half of the votes are received. This ensures that a sufficient number of nodes agree on the proposal, and even if a small number of nodes fail or send incorrect messages, it will not affect the overall decision. After confirming acceptance, the leader broadcasts the message to all nodes, informing other nodes to perform update operations and guiding them accordingly. Otherwise, if there are insufficient votes, the leader re-initiates the proposal or elects a new leader for a retry, with all nodes participating in the vote again to ensure uninterrupted consensus and consistency across any changes. According to the consensus proposal, the system specifies the data that needs to be updated, and each node will only update the parts marked as changed based on the consensus result, applying incremental data instead of performing an update on the entire dataset.

6. The method for adaptive data consistency in a power information system based on an incremental update mechanism according to claim 5, characterized in that, Before updating data, nodes use key attributes of resources in the distributed system to mark the power data. When multiple regional nodes operate on the same power data, decisions are made based on the weights of the key attributes. A distributed consensus algorithm ensures synchronization and consistency among nodes, and can select appropriate solutions in case of conflicts. Combined with an incremental update mechanism, only the changed power data is updated. Specifically, when detecting the synchronization and consistency of power data, the following consistency state equation is used to evaluate the synchronization status of power data, ensuring that inconsistencies are detected in a timely manner during the detection process: D t =aD t-1 +(1-a)D s , In the formula, D t D represents the power data status at the current point in time. t-1 D represents the power data status at the previous time point. s The data represents power synchronization data from other nodes. 'a' is a weighting factor of 0.5, indicating the relative importance of local power data and synchronization data. High network latency necessitates reliance on local data, increasing 'a'; low latency prioritizes synchronization data, decreasing 'a'. When |D t -D s When |>∈, the conflict resolution process is triggered, where ∈ is the allowable error threshold. The specific value needs to be determined comprehensively based on power business requirements, data characteristics, and system tolerance. If based on load: In the formula ∈ 基线 The baseline allowable error threshold is set according to the criticality level of power data in this invention, such as key fields ∈ 基线 =0, non-key fields ∈ 基线 >0, If based on data type: Key power parameters are set to ∈=0, and non-key monitoring data are set to ∈=+∞.

7. The adaptive data consistency maintenance method for power information systems based on incremental update mechanism according to claim 6, characterized in that, User-submitted updates to power data objects or self-maintenance updates to system information include updates to key attributes, which involve the following confirmation process: The set of all key attributes of a type of power resource is called the key attribute set of that type of resource. The key attribute set of power resource Res is denoted as KS(Res). OCKP uses semantic representation for updates, describing each modification in the update process separately, denoted as Updatei,j(DE). Where DE is the data object identifier, i is the update publishing node ID, and j is the update sequence number. It is assumed that the data is allocated on a per-node basis. Updatei,j(DE) is a binary tuple<Ii,j,k(DE),Ui,j,k(DE)> The set, where Ii,j,k(DE) is the description of the updated data information of data object DE, Ui,j,k(DE) is the corresponding information after the data object is modified, and k is the number of the modified item. If the update Updatei,j(DE) satisfies: Therefore, updating Updatei,j(DE) is considered a critical update to the data object DE; The power data migration and load balancing are dynamically adjusted using a heat-based algorithm. The access frequency of each power data block is statistically analyzed, and a heat value H is generated. i Represented as: In the formula, T represents the time window; The popularity score can be dynamically weighted based on access time and access type to refine decision-making and improve adaptability. H′ i =H i ×α 时间 ×β 类型 , Among them, H′ i For the weighted heat value, α 时间 β is the time weighting coefficient. 类型 For type weight coefficients, Based on the popularity level of the power data, data blocks with popularity values ​​above a threshold are identified as hot power data, while data blocks with popularity values ​​below the threshold are identified as cold data. Hot power parameters are processed using the strong consistency protocol Raft and stored on high-performance storage media, while low-frequency monitoring data are processed using the eventual consistency protocol Dynamo and stored on low-cost media. The threshold is dynamically adjusted based on system status and access mode: If based on a time window, the access frequency is counted in different time windows to adjust the threshold: Threshold=μ+k·σ, In the formula, μ is the average access frequency, k is an adjustment coefficient ranging from 1 to 3, and σ is the standard deviation of the access frequency. To ensure the system's dynamic adaptability and decision-making accuracy, μ and σ need to be recalculated periodically, and the thresholds need to be dynamically updated, with a calculation time set to once every 5 minutes. If based on load, when the system load is high, the threshold is lowered to migrate more hot data; conversely, the threshold is raised to reduce migration overhead when the system load is low. Using the popularity value as the weight, for each data block D i Calculate its migration priority: P i =H i ·C i , In the formula C i It's the migration cost. Based on priority sorting from high to low, the highest priority power data blocks are selected for migration. If a node fails, the priority is also used to prioritize the recovery of hotspot power data. Using heat value and node load as weights, the load distribution is dynamically adjusted for each node N. j Calculate its load weight: In the formula Load(N) j ) represents node N j Current load status; load can be seen as a comprehensive indicator of CPU utilization, memory consumption, and network bandwidth. Based on the above heat values ​​and load weights, for each power data block D i Calculate the probability of its access request allocation: In the formula, P ij Represents power data block D i The access request is assigned to node N. j The probability, ∑ k H i ·W k Calculate the sum of the heat weights of all nodes for standardization.

8. The adaptive data consistency method for power information systems based on incremental update mechanism according to claim 1, characterized in that, The feedback process involves deploying monitoring agents on regional nodes of the distributed system. These agents are configured with appropriate sampling rates and use the Prometheus monitoring tool to monitor key performance indicators of each node in real time, such as CPU utilization, memory usage, and network traffic. The Elastic Stack collects and analyzes power system logs and event logs, periodically collecting this local state data and pushing information to the central monitoring system. For key power indicators, the sampling rate can be set to once per second or once every two seconds; for indicators that are updated less frequently, it can be set to once per minute; and for log data, it can be set to once per minute to once per hour. The central monitoring system uses the 3-Sigma rule in anomaly detection algorithms to analyze the collected data. Based on the assumption of normal distribution, it identifies power data exceeding the range of μ±3σ as abnormal data. Abnormal conditions: x < μ - 3σ or x > μ + 3σ The mean μ is the center of the distribution, and the standard deviation σ measures the degree of dispersion of the data. When the indicators exceed the preset range, the system can automatically identify the abnormal state, start the recovery process, re-elect a leader, or re-synchronize the power data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements an adaptive data consistency maintenance method for a power information system based on an incremental update mechanism, as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, the computer instruction implements an adaptive data consistency maintenance method for power information systems based on an incremental update mechanism as described in any one of claims 1-8.

Images