A consensus-driven multi-state multi-application real-time synchronization method and system

Abstract

The application discloses a kind of based on consensus driving's polymorphism multi-application real-time synchronization method and system, it is related to power automation system technical field, the method includes: from candidate node group elects only main node, and the rest loading same type combination node is as slave node;Real-time change data is written into real-time database and local cache queue simultaneously;Reading, collection and compression processing are carried out to the newly added real-time change data, and data packet is generated;Slave node generates aggregation verification information for integrity verification from the data packet belonging to the same combination;After main node receives, local aggregation verification information is calculated and compared;Main node legal and effective data packet is directionally forwarded to all other slave nodes loaded same combination in local area network.This application realizes the fine granularity isolation of synchronization link by the double-dimension independent election of main node according to running state and application, effectively avoids the performance bottleneck and single-point failure risk under traditional single-main node architecture.

Description

Technical Field

[0001] This invention relates to the field of power automation system technology, and in particular to a consensus-driven, multi-state, multi-application real-time synchronization method and system. Background Technology

[0002] With the rapid development of new power systems, they exhibit significant characteristics such as diversified power supply structures, complex load characteristics, and intelligent operation control, placing higher demands on data processing and applications. In different business scenarios such as daily operation, planning research, and testing verification of power systems, the system needs to be able to flexibly switch to different operating states, i.e., multi-mode operation. For example, the real-time mode is mainly used to support real-time monitoring and dispatching of the power system, requiring extremely high timeliness in data processing and response; the research mode focuses on in-depth mining of historical data, simulation analysis, and verification of new algorithms, with stringent requirements for data integrity and accuracy; the testing mode is used for the access testing of new equipment and functions, requiring simulation of various operating conditions and ensuring the stability of data interaction. Meanwhile, in these different modes, there are numerous diverse applications, such as terminal applications responsible for real-time data acquisition, computing applications performing complex data processing and analysis, and public applications providing basic service support. These applications are often distributed across different nodes within a local area network, each generating, processing, and storing large amounts of real-time data.

[0003] Against this backdrop, data, as a core element of power system operation, directly impacts the accuracy of system decisions and the reliability of operation through its synchronization efficiency and consistency across different nodes and applications. Traditional data synchronization technologies have gradually revealed numerous shortcomings when facing complex scenarios with multiple modes and applications. Existing technical solutions most similar to this invention mainly include synchronization technologies based on traditional master-slave replication and asynchronous communication technologies based on simple message queues. Synchronization technologies based on traditional master-slave replication typically involve setting up one master node and multiple slave nodes, with the master node actively synchronizing its updated data to the slave nodes.

[0004] However, in multi-application scenarios where multiple nodes may generate data updates, this approach can easily lead to the master node becoming a bottleneck. Furthermore, if the master node fails, the availability of the entire synchronization system will be severely impacted, making it difficult to meet the requirements for data consistency and high availability under multi-state switching. Its synchronization latency is also often difficult to control within milliseconds. While asynchronous communication technology based on simple message queues achieves data transmission through message publish-subscribe mechanisms, although it decouples direct dependencies between applications to some extent, the lack of an effective consensus mechanism makes it difficult to guarantee the order and consistency of message transmission. Especially during network congestion or node failures, problems such as data loss, duplication, or out-of-order delivery can easily occur, leading to inconsistent application data across nodes and failing to meet the stringent requirements of strong data consistency in power systems.

[0005] In addition, most of these existing technologies are designed for a single form or specific application scenario, lacking overall consideration for dynamic application switching and data consistency across the entire link in multi-mode scenarios, making it difficult to adapt to the complex requirements of multi-mode and multi-application collaborative operation of new power systems. Summary of the Invention

[0006] Therefore, it is necessary to provide a consensus-driven, multi-application real-time synchronization method and system to address the aforementioned technical issues.

[0007] In a first aspect, the present invention provides a consensus-driven, multi-state, multi-application real-time synchronization method, comprising:

[0008] S1. Based on the combination formed by matching a single application with a single running state, elect a unique master node from the candidate node group, and load the remaining nodes with the same type of combination as slave nodes.

[0009] S2. When an application on any node generates real-time changed data, the real-time changed data is simultaneously written to the real-time database and the local cache queue; and the local cache queue is continuously monitored to read, collect, and compress the newly added real-time changed data to generate data packets.

[0010] S3. The slave node generates aggregated verification information for integrity verification for data packets belonging to the same group, and sends the data packets and aggregated verification information together to the master node corresponding to the group. After receiving the data packets, the master node calculates the local aggregated verification information and compares them. If they match, the data is deemed legal and valid. If they do not match, an exception recovery process is triggered.

[0011] S4. The master node forwards the valid data packets to all other slave nodes in the local area network that have the same combination. After receiving the data packets, each slave node writes the real-time changes in the data packets into the real-time database, thus completing an incremental synchronization.

[0012] Furthermore, based on the combination formed by matching a single running state with a single application, a unique master node is elected from the candidate node group, and the remaining nodes loaded with the same type of combination are designated as slave nodes, including:

[0013] S11. Perform a full verification and veto process on each candidate node in the candidate node group. If any verification item does not meet the preset screening threshold, the candidate node is vetoed.

[0014] S12. If the list of available nodes after filtering is empty, expand the candidate node group and restart the initial screening process; if only one candidate node remains after filtering, determine that candidate node as the main node of the current running state and application combination; if multiple candidate nodes remain after filtering, execute S13.

[0015] S13. Calculate the cumulative load value based on the baseline load weight of each primary node role and count all primary node roles already assumed by the candidate node. If the cumulative load value exceeds the preset resource threshold of the candidate node, the candidate node loses its eligibility for this round and is selected from the remaining candidate nodes.

[0016] S14. For each candidate node that passes the initial screening, obtain the synchronization operation index of the candidate node for the current combination. If the synchronization operation index exceeds the preset index threshold, the candidate node will be vetoed in this round of election. The synchronization operation index includes the backlog depth of the window to be confirmed.

[0017] S15. Obtain the operating indicators of the remaining candidate nodes after screening, calculate the comprehensive score according to the preset weighting formula, and determine the candidate node with the highest score as the master node of the current running state and application combination; if the comprehensive scores are the same, randomly select any candidate node as the master node.

[0018] Furthermore, the slave node generates aggregated verification information for integrity verification from data packets belonging to the same group, and sends the data packets and aggregated verification information together to the master node corresponding to that group, including:

[0019] S31. Based on the sliding window mechanism, according to the preset time window length or data volume window size, multiple real-time changing data belonging to the same running state and application are aggregated into a data packet.

[0020] S32. Preset differentiated window parameter baselines according to the type of operating state, and dynamically adjust the sliding window size in real time according to the current network operating indicators, and the adjustment range of the sliding window size is constrained within the parameter boundaries of the corresponding operating state.

[0021] S33. The Merkle tree algorithm is used to calculate and generate unique aggregated verification information for the data packet; the aggregated verification information serves as the digital fingerprint of the data packet.

[0022] S34. The aggregated verification information is used as the data packet header, combined with the compressed data packet to form a complete message, and the complete message is sent to the corresponding master node through the dedicated independent transmission control protocol (TCP) link of the current running state and the application. The data packet header contains the total Merkle root hash and the sub-Merkle root hash, micro-window index and semantic tag mapping table.

[0023] Furthermore, the Merkle tree algorithm is used to calculate and generate unique aggregated check information for the data packets, including:

[0024] S331. Extract business semantic features from all real-time change data within the sliding window, and generate a multi-dimensional semantic feature vector and business semantic label corresponding to each piece of real-time change data.

[0025] S332. Based on the semantic association between multidimensional semantic feature vectors and business semantic tags, perform clustering with capacity constraints to obtain several semantic clusters; each semantic cluster corresponds to a micro-window, and the real-time changing data within the same semantic cluster has business logical associations;

[0026] S333. Calculate the submerkel root hash for all real-time changing data in each micro-window;

[0027] S334. Generate the total Merkle root hash of the next level as the aggregated verification information by using all the child Merkle root hashes as leaf nodes.

[0028] Furthermore, based on the semantic relationships between multidimensional semantic feature vectors and business semantic tags, clustering with capacity constraints is performed, resulting in several semantic clusters, including:

[0029] S3321. The optimization objective is to maximize the mutual information between the micro-window partitioning result and the data business semantic tags, and the preset lower and upper limits of the number of data entries in the micro-window are used as constraints.

[0030] S3322. Initialize each micro-window as an empty cluster, randomly select a real-time changing data as the seed data of the first micro-window, and then distribute the remaining real-time changing data to the nearest micro-window according to the current semantic similarity to generate an initial partitioning scheme.

[0031] S3323. Calculate the mutual information between each micro-window and the corresponding data service semantic tag under the current partitioning scheme. If the mutual information is lower than the preset threshold, perform the micro-window splitting operation: separate the data subset with the lowest mutual information contribution from the current micro-window to form a new micro-window.

[0032] S3324. Iteratively execute data migration attempts between micro-windows, recalculate mutual information after each migration. If the mutual information increases after migration and does not violate capacity constraints, retain the migration result; if the mutual information no longer increases or reaches the maximum number of iterations in multiple consecutive iterations, terminate the iteration.

[0033] S3325. The partitioning result with the largest mutual information at the end of the iteration is taken as the micro-window partitioning result.

[0034] Furthermore, the micro-window splitting operation includes:

[0035] S33231. Calculate the marginal contribution value of each real-time changing data in the current micro-window to the mutual information content; the marginal contribution value is the decrease in the mutual information content between the micro-window and the business semantic tag after removing the data;

[0036] S33232: Mark data whose marginal contribution value is lower than the preset splitting threshold as splittable data, remove splittable data from the current micro-window, and form a new candidate micro-window;

[0037] S33233. Check whether the number of real-time change data entries in the newly formed candidate micro-window is lower than the preset lower limit. If it is lower than the lower limit, cancel the splitting operation and redistribute the splittable data to the adjacent micro-window with the least mutual information loss.

[0038] Furthermore, the aggregated verification information is used as the data packet header, combined with the compressed data packet to form a complete message, and then sent to the corresponding master node through the dedicated independent transmission control protocol link between the current running state and the application.

[0039] S341. The slave node combines the currently combined complete message with the dependency sequence number snapshot array carried in the cache data structure (CacheRecord), encapsulates it in the extended header of the complete message, and sends it to the master node.

[0040] S342. After the master node receives the complete message and completes consensus verification, it parses the dependency sequence number snapshot array in the extended packet header and compares each dependency sequence number in the dependency sequence number snapshot array with the latest globally confirmed sequence number of the corresponding running state and the corresponding application recorded on the local machine.

[0041] S343. If any dependent sequence number is found to be lagging behind the current global progress by more than a preset threshold, a dependency barrier instruction is appended to the extended field of the Transmission Control Protocol (TCP) acknowledgment response message returned to the slave node. The dependency barrier instruction includes the lagging dependency runtime identifier, the dependency application identifier, and the target global sequence number to be filled in.

[0042] S344. After receiving the Transmission Control Protocol (TCP) acknowledgment response with the Dependency Fence instruction from the slave node, the incremental synchronization process of the current combination is paused. The slave node first requests the missing real-time change data between the current local sequence number and the target global sequence number from the master node. After the missing data is filled in, the incremental synchronization processing of the current sliding window is resumed.

[0043] Furthermore, the master node forwards valid data packets to all other slave nodes within the local area network that load the same combination. Each slave node, upon receiving the packet, writes the real-time changes in the data packet into the real-time database, completing one incremental synchronization cycle, which includes:

[0044] S41. After receiving the complete message, the master node extracts the aggregated verification information from the data packet header of the complete message and parses the compressed data packet in the complete message.

[0045] S42. The master node uses the same Merkle tree algorithm as the slave node to calculate local aggregate check information for the decompressed data packets.

[0046] S43. Compare the local aggregated verification information with the aggregated verification information sent by the slave node;

[0047] S44. If the total Merkle root hash matches, the data packet is determined to be complete and tamper-proof, and receipt is confirmed.

[0048] S45. If the comparison is inconsistent, the master node locates the target micro-window with a hash mismatch based on the data packet header, requests the slave node to resend the data subset corresponding to the target micro-window, and performs a comparison item by item; if the resent data verification passes, the data packet is determined to be valid as a whole.

[0049] S46. If the number of failed retransmissions exceeds a preset threshold, disconnect from the slave node, wait for the slave node to reconnect, and then send the full real-time database profile data to the slave node for recovery.

[0050] Furthermore, the master node locates the target micro-window with a hash mismatch based on the data packet header, requests the slave node to retransmit the data subset corresponding to the target micro-window, and compares it line by line; if the retransmitted data verification passes, the data packet is determined to be valid overall, including:

[0051] S451. The master node compares the submerkle root hash of each micro-window with the submerkle root hash, micro-window index and semantic tag mapping table in the data packet header one by one, and locates the target micro-window index of all hash mismatches.

[0052] S452. The master node sends a location request back to the slave node. The location request includes a list of target micro-window indices. After receiving the location request, the slave node repackages the original data subset corresponding to the target micro-window index and sends it to the master node.

[0053] S453. The master node compares each retransmitted data subset to locate the erroneous data entry; if the retransmitted data passes the verification, the data packet is deemed valid as a whole.

[0054] Secondly, the present invention also provides a consensus-driven, multi-state, multi-application real-time synchronization system, the system comprising:

[0055] The master-slave node election module is used to elect a unique master node from the candidate node group based on the combination formed by matching a single application with a single running state, and to load the remaining nodes with the same type of combination as slave nodes.

[0056] The data capture module is used to write real-time changed data to both the real-time database and the local cache queue when the application on any node generates real-time changed data; and to continuously monitor the local cache queue, read, collect and compress the newly added real-time changed data, and generate data packets.

[0057] The consensus verification module is used by slave nodes to generate aggregated verification information for integrity verification of data packets belonging to the same group, and send the data packets and aggregated verification information together to the master node corresponding to the group. After receiving the data packets, the master node calculates the local aggregated verification information and compares it. If they match, the data is determined to be legal and valid. If they do not match, anomaly recovery processing is triggered.

[0058] The targeted distribution synchronization module is used by the master node to forward legitimate and valid data packets to all other slave nodes in the local area network that have the same combination of data. After receiving the data packets, each slave node writes the real-time changes in the data packets into the real-time database, thus completing an incremental synchronization.

[0059] The beneficial effects of this invention are as follows:

[0060] 1. This invention achieves fine-grained isolation of the synchronization link by independently electing the master node based on both the runtime state and the application. Different runtime states and different business applications each have independent master nodes and dedicated transmission links. Synchronization anomalies or master node failures in any combination are isolated within that combination and will not affect the synchronization process of other runtime states or other business applications. Simultaneously, the same physical node can act as a master node or a slave node in different combinations depending on its loaded state and application, achieving flexible reuse and load balancing of node resources and effectively avoiding the performance bottlenecks and single-point-of-failure risks of traditional single-master-node architectures.

[0061] 2. This invention significantly reduces the interaction overhead of consensus verification through a batch consensus verification mechanism based on a sliding window. Slave nodes aggregate multiple real-time changing data from the same runtime state and application into data packets, generating aggregated verification information on a packet-by-packet basis. The master node only needs to verify the data packet once to confirm the integrity of the batch data, eliminating the need for individual verification. Simultaneously, the transmission links for each runtime state and application are independent, allowing data to be transmitted in parallel on each link without interference. This avoids transmission congestion and latency accumulation caused by mixed data, effectively improving transmission efficiency in multi-state, multi-application concurrent synchronization scenarios.

[0062] 3. This invention ensures eventual consistency of data synchronization through a multi-level anomaly recovery mechanism. When verification fails, the master node accurately locates the range of erroneous data based on the micro-window index mapping table and performs incremental compensation, significantly reducing the recovery cost. Full-scale cross-section recovery is only triggered when incremental compensation repeatedly fails, avoiding excessive consumption of system resources by frequent full-scale synchronization. Slave nodes automatically detect connection anomalies through a heartbeat mechanism and initiate reconnection. After reconnection, they automatically receive full-scale cross-section data and restore it to a consistent state, ensuring that data from each node can still be quickly restored to consistency in abnormal scenarios such as node restarts and network outages, meeting the stringent requirements of power systems for data reliability. Attached Figure Description

[0063] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0064] Figure 1 This is a flowchart of a consensus-driven, multi-application real-time synchronization method according to an embodiment of the present invention;

[0065] Figure 2 This is a consensus-driven, multi-application, synchronization architecture diagram according to an embodiment of the present invention;

[0066] Figure 3 This is a diagram showing the relationship between states, applications, and nodes according to an embodiment of the present invention;

[0067] Figure 4 This is a flowchart of the synchronization forwarding process according to an embodiment of the present invention;

[0068] Figure 5 This is a flowchart of the master-slave node data forwarding process according to an embodiment of the present invention;

[0069] Figure 6 This is a system principle block diagram of a consensus-driven, multi-state, multi-application real-time synchronization system according to an embodiment of the present invention.

[0070] The diagram shows the following modules: 1. Master-slave node election module; 2. Data capture module; 3. Consensus verification module; 4. Targeted distribution and synchronization module. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0072] Please see Figure 1 This provides a consensus-driven, multi-application, real-time synchronization method, including:

[0073] S1. Based on the combination formed by matching a single application with a single running state, elect a unique master node from the candidate node group, and load the remaining nodes of the same type as slave nodes.

[0074] Specifically, such as Figure 2 The diagram shown illustrates a consensus-driven, multi-state, multi-application real-time synchronization architecture. After data is received, it is first synchronized to the synchronization master node where the application in that state resides, based on the data's running state (hereinafter referred to as state) and the application. The master node then forwards the data to other nodes via TCP. During the forwarding process from slave nodes to the master node, the slave nodes and the master node perform consensus verification on the data packets to ensure the legitimacy of the nodes and the validity of the data.

[0075] The synchronization master node is generated from the fault-tolerant candidate node group. Each application selects a unique synchronization master node. The master node is selected by the arbitration service based on the machine running status, machine loading status, application type, and process running status within the group.

[0076] The arbitration service uses a full verification and veto mechanism to screen available nodes: it verifies the machine's running status, loading status, and the corresponding application process status one by one, and removes nodes that do not meet the threshold; if the list of available nodes is empty after screening, the candidate node group is expanded and the election process is restarted; if only one node remains, it directly enters the fault tolerance verification stage; if multiple nodes remain, it enters the quantitative scoring stage.

[0077] Since a node may load different applications in multiple states, the synchronization master node for different applications may be the same node. Figure 3 This is a graph showing the relationship between states, applications, and nodes. States provide data for different times and scenarios to the system, while applications provide logical support to the system.

[0078] Real-time data generated by a certain application in a certain state is written to the local cache queue while being written to the real-time database. The consensus-driven multi-state multi-application real-time data synchronization module continuously monitors the local cache queue. When data is generated, the synchronization module captures the changed data from the queue, compresses the data, and forwards it to the master node of the application in that state. If the data is control operation data, it will be encrypted and forwarded.

[0079] In the forwarding of multi-state, multi-application real-time data, each data link is independent of the others, avoiding mutual interference between data of different states and applications during forwarding, and ensuring the correctness and independence of various types of data within the transmission channel. This link isolation design allows data of different states (such as real-time state, research state, and test state) and different applications (data processing applications, data acquisition applications, public infrastructure applications, etc.) to be transmitted in parallel in their respective independent channels, effectively avoiding problems such as transmission delays, out-of-order data packets, or information leakage caused by data mixing.

[0080] Furthermore, each node may load different states and applications, such as Figure 4 As shown, node A loads R_S and R_P (real-time data processing application and real-time common basic application), while node B loads R_S and R_F (real-time data processing application and real-time data acquisition application). If A or B directly sends real-time changing data to the other party, the other party will be unable to process real-time data of type R_P or R_F, resulting in an exception when writing to the real-time database, thus affecting data synchronization performance.

[0081] This invention does not directly synchronize real-time change data generated by node A to other nodes within the local area network. Instead, it first determines the master node of the synchronization module corresponding to the application loaded on this node. For example, if node M1 is the master node of R_S (real-time data processing application), node A will first forward the real-time data of R_S to node M1, and then node M1 will forward it to node B and node M2 ​​(other nodes loaded with real-time data processing applications). The transmission links between different states and applications are independent and use TCP communication. The master node of different applications in different states may be the same node. The detailed process of master selection and forwarding is as follows: Figure 3 As shown, each node loads different states and applications.

[0082] In the description of this invention, based on the combination formed by matching a single running state with a single application, a unique master node is elected from the candidate node group, and the remaining nodes loaded with the same type of combination are designated as slave nodes, including:

[0083] S11. Perform a full verification and veto process on each candidate node in the candidate node group. If any verification item does not meet the preset screening threshold, the candidate node is vetoed.

[0084] S12. If the list of available nodes after filtering is empty, expand the candidate node group and restart the initial screening process; if only one candidate node remains after filtering, determine that candidate node as the master node of the current running state and application combination. If multiple candidate nodes remain after filtering, execute S13.

[0085] Specifically, if the list of available nodes is empty after screening, the arbitration service expands the candidate range from the fault-tolerant candidate node group and restarts the election process. If only one candidate node remains, it directly enters the fault tolerance verification stage and determines that node as the master node of the "state + application" combination. If multiple candidate nodes remain, it enters the quantitative scoring stage.

[0086] S13. Calculate the cumulative load value based on the baseline load weight of each primary node role, considering all primary node roles already assumed by the candidate node. If the cumulative load value exceeds the preset resource threshold for the candidate node, the candidate node loses its eligibility for this round, and a replacement node is selected from the remaining candidate nodes.

[0087] Specifically, since a physical node may load multiple applications in different states simultaneously, and the synchronization master node for different applications may be the same node, it is necessary to count all synchronization master roles already undertaken by the candidate node and calculate the cumulative load value according to the baseline load weight of each role. If the cumulative load value exceeds the node's preset resource threshold, the node will lose its eligibility for this round of candidate status, in order to avoid a single node becoming a performance bottleneck due to undertaking too many master roles.

[0088] S14. For each candidate node that passes the initial screening, obtain the synchronization operation index of the candidate node for the current combination. If the synchronization operation index exceeds the preset index threshold, the candidate node will be vetoed in this round of election. The synchronization operation index includes the backlog depth of the window to be confirmed.

[0089] Specifically, for each candidate node that passes the initial screening, the synchronization module operation metrics for that node in relation to the current state and application combination are obtained. These metrics include the difference in the maximum window sequence number that has been sent but has not yet received an acknowledgment response from the master node within the current sliding window, i.e., the backlog depth of the unacknowledgment window. If the backlog depth of the unacknowledgment window exceeds a preset threshold, the node is vetoed in this round of election to prevent a node with a severe backlog from being elected as the master node and further exacerbating the synchronization delay.

[0090] S15. Obtain the operating indicators of the remaining candidate nodes after screening, calculate the comprehensive score according to the preset weighting formula, and determine the candidate node with the highest score as the master node of the current running state and application combination; if the comprehensive scores are the same, randomly select any candidate node as the master node.

[0091] Specifically, the overall score The quantitative scores are as follows:

[0092] ;

[0093] The definitions of each indicator are as follows:

[0094] ;

[0095] ;

[0096] ;

[0097] In the formula, This is the normalized value of the reciprocal of CPU utilization. This is the normalized value of the inverse of memory utilization. The CPU utilization of the candidate nodes. For the memory utilization of candidate nodes, T represents the continuous runtime of the candidate nodes. min and T max These are the minimum and maximum runtime values, respectively. The identification coefficients for whether the candidate node loads the target state c and the target application a.

[0098] S2. When an application on any node generates real-time changing data, the real-time changing data is simultaneously written to the real-time database and the local cache queue; and the local cache queue is continuously monitored to read, collect, and compress the newly added real-time changing data to generate data packets.

[0099] Specifically, during data access, the application (one or more applications in different states) first writes the data to the real-time database and local cache queue of the local node. Then, the synchronization module of the local node forwards the data to the master node of the corresponding state and application based on the state number and application number of the captured data. The data format CacheRecord in the queue is as follows:

[0100] “struct CacheRecord{

[0101] ACE_UINT64 obid;

[0102] AType atype;

[0103] ACE_UINT32 write_option:16;

[0104] ACE_UINT32 flag:16;

[0105] ACE_UINT32 source_machine:16;

[0106] ACE_UINT32 transit_machine:16;

[0107] ACE_UINT32 dbDataType:8;

[0108] ACE_UINT32 ftack_ft:8;

[0109] ACE_UINT32 ftack_provider:8;

[0110] ACE_UINT32 ftack_notif:8;

[0111] ACE_UINT32 ctx_no;

[0112] ACE_UINT32 app_no;

[0113] ACE_UINT64 oper;

[0114] RecordValue old_value;

[0115] RecordValue value;

[0116] RecordTime seq_time;

[0117] };”

[0118] The data format includes ID number, attribute type, value, timestamp, status number, application number, the machine that wrote the data, and the machine to which it was sent.

[0119] To improve forwarding performance, the forwarding module compresses multiple data packets in the queue according to their state and application before sending them. Meanwhile, to reduce the impact of encryption / decryption on transmission performance, the synchronization module filters out real-time control data based on obid and atype before encrypting and transmitting it.

[0120] Meanwhile, the forwarding module performs packet assembly and compression on the captured data. After comparing algorithms such as LZ4, gzip, and QuickLZ, the zstd algorithm exhibits the best overall performance in terms of compression speed, decompression speed, and compression ratio, making it suitable for real-time TCP data transmission scenarios. Multiple tests determined compression level 2 to be the optimal parameter, balancing real-time performance with bandwidth gains. Finally, the zstd algorithm is integrated with the packet assembly process to achieve low-latency, high-compression-efficiency data processing, reducing the bandwidth pressure on the master node's forwarding.

[0121] S3. The slave node generates aggregated verification information for integrity verification for data packets belonging to the same group, and sends the data packets and aggregated verification information together to the master node corresponding to the group. After receiving the data packets, the master node calculates the local aggregated verification information and compares them. If they match, the data is determined to be legal and valid. If they do not match, the abnormal recovery process is triggered.

[0122] Specifically, consensus verification is a crucial step in ensuring the accuracy and consistency of data synchronization. Performing consensus verification on every single data packet would significantly impact the timeliness of real-time data. This invention performs consensus verification on batches of data packets within a time window during the data forwarding process from slave nodes to master nodes. When a slave node comes online, it sends a request to the corresponding master node based on its state and application. Upon receiving the request, the master node verifies the slave node's identity legitimacy, software environment compatibility, and initial configuration information. After successful verification, the master node assigns a unique session identifier to the slave node and establishes a secure data transmission channel.

[0123] In the description of this invention, the process of a slave node generating aggregated verification information for integrity verification from data packets belonging to the same group, and sending the data packets and aggregated verification information together to the master node corresponding to that group includes:

[0124] S31. Based on the sliding window mechanism, multiple real-time changing data belonging to the same running state and application are aggregated into a single data packet according to the preset time window length or data volume window size.

[0125] Specifically, a sliding window module is configured at the slave node. This module is used to define a data set within a preset time period or a preset data volume range. Based on the current runtime state and application type, the slave node aggregates multiple real-time changing data items belonging to the same runtime state and application combination into a single data packet, according to the preset time window length or data volume window size. During the aggregation process, the slave node ensures the integrity of the data packet and the original order of the data entries.

[0126] S32. Preset differentiated window parameter baselines according to the type of operating state, and dynamically adjust the sliding window size in real time according to the current network operating indicators, and the adjustment range of the sliding window size is constrained within the parameter boundaries of the corresponding operating state.

[0127] Specifically, different runtime states correspond to different window parameter baselines. In real-time mode, the window parameter baseline is configured as a low-latency-oriented small window mode; in research mode, it is configured as a high-throughput-oriented large window mode; and in test mode, it is configured as a high-reliability-oriented ultra-small window mode. During operation, the slave node dynamically adjusts the sliding window size in real time based on at least one of the following runtime metrics: current network round-trip latency, master node CPU load, and data generation rate. Each adjusted sliding window size is constrained within the preset parameter boundaries for the corresponding runtime state to prevent the window size from exceeding the adaptation range.

[0128] S33. The Merkle tree algorithm is used to calculate and generate unique aggregated check information for the data packet. The aggregated check information serves as the digital fingerprint of the data packet.

[0129] Specifically, the aggregation verification information generation module of the slave node uses the Merkle tree algorithm to calculate and generate unique aggregation verification information for the data packet. This aggregation verification information serves as the digital fingerprint of the data packet, comprehensively reflecting the characteristics of all real-time changing data within the packet. The synchronization master node verifies this aggregation verification information to determine whether the data within the data packet has been entirely tampered with or lost.

[0130] In the description of this invention, the Merkle tree algorithm is used to calculate and generate unique aggregated verification information for the data packets, including:

[0131] S331. Extract business semantic features from all real-time change data within the sliding window, and generate a multi-dimensional semantic feature vector and business semantic label corresponding to each piece of real-time change data.

[0132] Specifically, each slave node performs business semantic feature extraction on all real-time change data within the sliding window. Each piece of real-time change data is stored in a CacheRecord structure, which includes a unique data identifier, attribute type, value before change, value after change, timestamp, running state identifier, application identifier, local node identifier, and target node identifier. The slave node extracts at least one of the following from the CacheRecord structure as a business semantic attribute: device identifier, control loop number, and section identifier. The slave node then performs numerical mapping on the extracted business semantic attributes to generate a multi-dimensional semantic feature vector corresponding to each piece of real-time change data, and uses the device type and loop category from the business semantic attributes as business semantic labels.

[0133] S332. Based on the semantic relationships between multi-dimensional semantic feature vectors and business semantic tags, clustering with capacity constraints is performed to obtain several semantic clusters. Each semantic cluster corresponds to a micro-window, and the real-time changing data within the same semantic cluster has business logical relationships.

[0134] Specifically, the slave node uses the cosine similarity between the multidimensional semantic feature vectors and the matching degree between the business semantic tags as the metric for semantic association, and determines real-time change data with the same device identifier, the same control loop number, or the same cross-section identifier as having semantic association. Based on the semantic association, the slave node performs capacity-constrained clustering on all real-time change data within the sliding window, and groups real-time change data with semantic association into the same semantic cluster.

[0135] Each semantic cluster corresponds to a micro-window, and the real-time changing data within the same micro-window exhibits cohesion in terms of business semantics. The number of data entries in each micro-window is constrained by a preset lower limit and an upper limit. The preset lower limit is used to prevent the micro-window from being too small, which would increase the verification overhead, and the preset upper limit is used to prevent the micro-window from being too large, which would result in excessively high retransmission costs.

[0136] In the description of this invention, based on the semantic association between multidimensional semantic feature vectors and business semantic tags, a clustering with capacity constraints is performed to obtain several semantic clusters, including:

[0137] S3321. The optimization objective is to maximize the mutual information between the micro-window partitioning result and the data business semantic tags, and the preset lower and upper limits of the number of data entries in the micro-window are used as constraints.

[0138] Specifically, an information bottleneck clustering objective function is constructed from the nodes, with the optimization objective being to maximize the mutual information between the micro-window partitioning result and the data business semantic tags, so that the micro-window partitioning result retains the business semantic information of the original data to the greatest extent. The formula for calculating the mutual information is as follows:

[0139] ;

[0140] In the formula, X represents the micro-window partitioning result, Y represents the data business semantic label, H(Y) is the information entropy of the business semantic label, and H(Y|X) is the conditional entropy of the business semantic label under the known micro-window partitioning conditions. The greater the mutual information, the higher the degree to which the micro-window partitioning result preserves the business semantic label. Simultaneously, the number of data entries in the micro-window is within a preset lower limit L. min With the preset upper limit value L max The constraints between these conditions ensure that the number of data entries in each micro-window meets the preset capacity requirements.

[0141] S3322. Initialize each micro-window as an empty cluster, randomly select a real-time changing data as the seed data of the first micro-window, and then distribute the remaining real-time changing data to the nearest micro-window according to the current semantic similarity to generate an initial partitioning scheme.

[0142] Specifically, each micro-window is initialized as an empty cluster by the node. The node randomly selects one real-time change data point from all real-time change data within the sliding window as the seed data for the first micro-window, and assigns the seed data to the first micro-window.

[0143] Starting from the node, the remaining real-time changing data are traversed sequentially. The semantic similarity between the current data and each existing micro-window is calculated, and the current data is assigned to the micro-window with the highest semantic similarity. After all real-time changing data has been assigned, the initial partitioning scheme is obtained from the node. The semantic similarity is calculated as the cosine similarity between the multidimensional semantic feature vector of the current data and the center vector of the micro-window, where the center vector of the micro-window is the arithmetic mean of the multidimensional semantic feature vectors of all data within the micro-window.

[0144] S3323. Calculate the mutual information between each micro-window and the corresponding data service semantic tag under the current partitioning scheme. If the mutual information is lower than the preset threshold, perform the micro-window splitting operation: separate the data subset with the lowest mutual information contribution from the current micro-window to form a new micro-window.

[0145] Specifically, the mutual information between each micro-window and its corresponding data business semantic tag under the current partitioning scheme is calculated from the nodes to evaluate the degree to which the current partitioning scheme retains business semantic information. The total mutual information of the current partitioning scheme is calculated as follows:

[0146] ;

[0147] In the formula, K is the total number of microwindows, and W k Let L be the data set within the k-th micro-window. k Let I(W) be the set of business semantic tags for the data within the k-th micro-window. k L k Let be the mutual information between the k-th micro-window and its internal data business semantic tags. When the total mutual information is lower than a preset threshold, the slave node determines that the current partitioning scheme fails to effectively preserve the business semantic information of the data and performs a micro-window splitting operation. The splitting operation is as follows: the slave node identifies the data subset within each micro-window that contributes the least to the mutual information, separates this data subset from the current micro-window, and forms a new micro-window based on the separated data subset to improve the semantic cohesion of the overall partitioning.

[0148] In the description of this invention, the micro-window splitting operation includes:

[0149] S33231. Calculate the marginal contribution value of each real-time changing data in the current micro-window to the mutual information content; the marginal contribution value is the decrease in mutual information content between the micro-window and the business semantic tag after the data is removed.

[0150] Specifically, during the micro-window splitting operation, the marginal contribution of each real-time changing data within the current micro-window to the mutual information is calculated from the node.

[0151] The i-th data d i Marginal contribution value to microwindow W The calculation method is as follows:

[0152] ;

[0153] in, This refers to the mutual information between the current micro-window W and the business semantic tag L. To remove the i-th data d from the micro-window W i Then, the mutual information between the micro-window and the business semantic label L. The higher the marginal contribution value, the more important the data is in maintaining the semantic cohesion of the micro-window; the lower the marginal contribution value, the weaker the semantic association between the data and other data in the micro-window.

[0154] S33232: Mark data with marginal contribution values ​​lower than the preset splitting threshold as splittable data, remove splittable data from the current micro-window, and form a new candidate micro-window.

[0155] Specifically, the slave node marks all real-time change data within the current micro-window whose marginal contribution value is lower than a preset splitting threshold as splittable data; the slave node removes the splittable data from the current micro-window as a whole, and forms a new candidate micro-window based on the splittable data.

[0156] S33233. Check whether the number of real-time change data entries in the newly formed candidate micro-window is lower than the preset lower limit. If it is lower than the lower limit, cancel the splitting operation and redistribute the splittable data to the adjacent micro-window with the least mutual information loss.

[0157] Specifically, the number of real-time change data entries for newly formed candidate micro-windows is checked at the node to see if it is lower than a preset lower limit value L. min If it is lower than L min The split operation is canceled from the node, and the splittable data is redistributed to the adjacent micro-window.

[0158] S3324. Iteratively execute data migration attempts between micro-windows. After each migration, recalculate the mutual information. If the mutual information increases after migration and does not violate the capacity constraint, retain the migration result. If the mutual information no longer increases or reaches the maximum number of iterations in multiple consecutive iterations, terminate the iteration.

[0159] Specifically, after completing the initial partitioning and splitting operations, the slave node iteratively performs migration attempts for data entries between different micro-windows. In each migration attempt, the slave node moves a data entry from its current micro-window to an adjacent micro-window, and recalculates the total mutual information after the migration.

[0160] When the total mutual information increases after migration and the number of data entries in the target micro-window does not exceed the preset upper limit, the slave node retains the migration result. When the total mutual information no longer increases in multiple consecutive iterations, or when the preset maximum number of iterations is reached, the slave node terminates the iteration.

[0161] S3325. The partitioning result with the largest mutual information at the end of the iteration is taken as the micro-window partitioning result.

[0162] Specifically, after a node terminates its iteration, it selects the partitioning scheme with the largest mutual information from all the partitioning schemes recorded during the entire iteration process, and uses this as the final micro-window partitioning result.

[0163] S333. Calculate the submerkel root hash for all real-time changes in each micro-window.

[0164] Specifically, for each micro-window obtained from the final partitioning of node S332, Merkle tree hash calculation is performed on all real-time changing data within the micro-window to generate the sub-Merkle tree root hash corresponding to the micro-window.

[0165] S334. Generate the total Merkle root hash of the next level as the aggregated verification information by using all the child Merkle root hashes as leaf nodes.

[0166] Specifically, each node takes the root hash of the child Merkle tree corresponding to all micro-windows as its leaf nodes, generates the hash of the parent node at the next higher level according to the Merkle tree construction rules, and calculates upwards layer by layer until the total Merkle tree root hash is obtained. The total Merkle tree root hash serves as the aggregate verification information for the data packet.

[0167] S34. The aggregated verification information is used as the data packet header, combined with the compressed data packet to form a complete message, and then sent to the corresponding master node through the dedicated independent transmission control protocol (TCP) link of the current running state and the application. The data packet header contains the total Merkle root hash and the sub-Merkle root hash, the micro-window index, and the semantic tag mapping table.

[0168] Specifically, the slave node uses the aggregated verification information (including the total Merkle root hash, the hashes of each sub-Merkle root hash, the micro-window index, and the semantic tag mapping table) as the data packet header, and combines it with the compressed data packet to form a complete message. In the micro-window index and semantic tag mapping table, each micro-window index is associated with and stores the sub-Merkle root hash corresponding to the micro-window, the original sequence number range of the data entries within the micro-window, and the business semantic tag corresponding to the micro-window. The slave node sends the complete message to the corresponding synchronization master node through an independent TCP link dedicated to the current running state and application combination. The TCP links corresponding to different running states and different application combinations are logically isolated from each other, and data transmission does not interfere with each other. The synchronization master node, through the semantic tag mapping table, directly determines the scope of business functions affected by the erroneous micro-window when locating it.

[0169] In the description of this invention, the aggregation verification information is used as the data packet header, combined with the compressed data packet to form a complete message, and the complete message is sent to the corresponding master node through the dedicated independent transmission control protocol link of the current running state and the application.

[0170] S341. The slave node combines the currently combined complete message with the dependency sequence number snapshot array carried in the cache data structure (CacheRecord), encapsulates it in the extended header of the complete message, and sends it to the master node.

[0171] Specifically, the slave node retrieves the dependency sequence number snapshot array carried in the CacheRecord structure. This dependency sequence number snapshot array records the sliding window global sequence numbers of the most recent successful synchronization with the synchronization master node for other specified runtime states and applications besides the current runtime state and application combination. The slave node encapsulates the dependency sequence number snapshot array in the extended header of the complete message, merges it with the complete message, and sends it to the synchronization master node.

[0172] S342. After the master node receives the complete message and completes consensus verification, it parses the dependency sequence number snapshot array in the extended packet header and compares each dependency sequence number in the dependency sequence number snapshot array with the latest globally confirmed sequence number of the corresponding running state and the corresponding application recorded on the local machine.

[0173] Specifically, after receiving the complete message, the synchronization master node first extracts the aggregated verification information from the message header and performs consensus verification. After successful consensus verification, the synchronization master node parses the dependency sequence number snapshot array in the extended packet header to obtain each dependency sequence number recorded in the array. The synchronization master node then compares each dependency sequence number with the latest globally confirmed sequence number of the corresponding running state and application recorded locally.

[0174] S343. If any dependent sequence number is found to be lagging behind the current global progress by more than a preset threshold, a dependency barrier instruction is appended to the extended field of the Transmission Control Protocol (TCP) acknowledgment response message returned to the slave node. The dependency barrier instruction includes the lagging dependency runtime identifier, the dependency application identifier, and the target global sequence number to be filled in.

[0175] Specifically, when the master node discovers that any dependent sequence number in the dependent sequence number snapshot array lags behind the latest globally confirmed sequence number of the corresponding runtime state and application recorded locally, and the lag exceeds a preset threshold, the master node determines that the slave node has a synchronization lag in the corresponding dependent runtime state or dependent application. To avoid data inconsistency across runtime states or applications, the master node appends a dependency barrier instruction to the extended field of the TCP acknowledgment response message returned to the slave node. The dependency barrier instruction includes the lagging dependent runtime state identifier, the dependent application identifier, and the target global sequence number to be supplemented.

[0176] S344. After receiving the Transmission Control Protocol (TCP) acknowledgment response with the Dependency Fence instruction from the slave node, the incremental synchronization process of the current combination is paused. The slave node first requests the missing real-time change data between the current local sequence number and the target global sequence number from the master node. After the missing data is filled in, the incremental synchronization processing of the current sliding window is resumed.

[0177] Specifically, after receiving the TCP acknowledgment response containing the dependency barrier instruction, the slave node parses the dependency barrier instruction to obtain the lagging dependency runtime identifier, dependency application identifier, and target global sequence number. The slave node pauses the incremental synchronization process of the current runtime and application combination and prioritizes requesting the missing real-time change data between the current local sequence number and the target global sequence number from the synchronization master node. After receiving and writing the missing real-time change data to complete the data filling, the slave node resumes the incremental synchronization processing of the current sliding window.

[0178] S4. The master node forwards the valid data packets to all other slave nodes in the local area network that have the same combination. After receiving the data packets, each slave node writes the real-time changes in the data packets into the real-time database, thus completing an incremental synchronization.

[0179] Specifically, the master node forwards the verified data packet, based on the runtime identifier and application identifier carried in the packet, through the independent TCP links corresponding to each slave node to all slave nodes within the local area network that have loaded the same runtime and application combination. After receiving the data packet, each slave node writes the real-time change data in the packet into its own real-time database, completing one incremental synchronization.

[0180] In the description of this invention, the master node forwards valid data packets to all other slave nodes within the local area network that load the same combination. Each slave node, upon receiving the packet, writes the real-time changes in the data packet into a real-time database, thus completing one incremental synchronization cycle.

[0181] S41. After receiving the complete message, the master node extracts the aggregated verification information from the data packet header of the complete message and parses the compressed data packet in the complete message.

[0182] Specifically, after receiving the complete message, the synchronization master node extracts aggregated verification information from the message header. This aggregated verification information includes the total Merkle root hash, the hashes of each sub-Merkle root hash, and a micro-window index-semantic tag mapping table. The synchronization master node then decompresses the compressed data packets in the complete message to obtain the decompressed data packets.

[0183] S42. The master node uses the same Merkle tree algorithm as the slave node to calculate local aggregate verification information for the decompressed data packets.

[0184] Specifically, the master node uses the same Merkle tree algorithm as the slave node to perform hash calculations on the decompressed data packets, generating local aggregated verification information. This local aggregated verification information includes the total Merkle tree root hash and the hashes of each sub-Merkle tree root obtained locally.

[0185] S43. Compare the local aggregated verification information with the aggregated verification information sent by the slave node.

[0186] Specifically, the master node compares the total Merkle root hash in the local aggregated verification information with the total Merkle root hash in the aggregated verification information sent by the slave node.

[0187] S44. If the total Merkle root hash matches, the data packet is determined to be complete and tamper-proof, and receipt is confirmed.

[0188] S45. If the comparison is inconsistent, the master node locates the target micro-window with the hash mismatch based on the data packet header, requests the slave node to resend the data subset corresponding to the target micro-window, and performs a line-by-line comparison. If the resent data verification passes, the data packet is determined to be valid as a whole.

[0189] Specifically, when the overall Merkle root hash comparison is inconsistent, the synchronization master node compares the sub-Merkle root hashes in the local aggregated verification information with those in the aggregated verification information sent by the slave nodes, based on the micro-window index and semantic tag mapping table, to locate all target micro-windows with hash mismatches. The synchronization master node sends a retransmission request to the slave nodes, requesting a retransmission of the original data subset corresponding to the target micro-window. After receiving the retransmitted original data subset, the synchronization master node performs a line-by-line comparison and verification on each real-time changed data item in the data subset. When the line-by-line comparison and verification passes, the synchronization master node determines that the data packet is valid as a whole.

[0190] In the description of this invention, the master node locates the target micro-window with a hash mismatch based on the data packet header, requests the slave node to retransmit the data subset corresponding to the target micro-window, and performs a line-by-line comparison. If the retransmitted data verification passes, the data packet is determined to be valid overall, including:

[0191] S451. The master node compares the submerkle root hash of each micro-window with the submerkle root hash of each micro-window in the data packet header, the micro-window index and the semantic tag mapping table, and locates the target micro-window index of all hash mismatches.

[0192] Specifically, the master node synchronizes and obtains the micro-window index and semantic tag mapping table. This mapping table contains each micro-window index, the corresponding sub-Merkel root hash of each micro-window, and the corresponding semantic tag of each micro-window. The master node compares each sub-Merkel root hash calculated locally with the corresponding sub-Merkel root hash sent by the slave node one by one, marks all micro-windows with mismatched hashes, and obtains the corresponding micro-window index list.

[0193] S452. The master node sends a location request back to the slave node, which includes a list of target micro-window indices. After receiving the location request, the slave node repackages the original data subset corresponding to the target micro-window indices and sends it back to the master node.

[0194] Specifically, the master node sends a location request back to the slave node, the location request containing the target micro-window index list obtained in S451. After receiving the location request, the slave node extracts the original real-time change data subset corresponding to each target micro-window index from its own cache queue according to the target micro-window index list, repackages it, and sends it to the master node.

[0195] S453. The master node compares each retransmitted data subset to identify erroneous data entries. If the retransmitted data passes verification, the entire data packet is deemed valid.

[0196] Specifically, after receiving the retransmitted data subset, the synchronization master node compares each real-time change data entry in the subset one by one to accurately locate any erroneous real-time change data entries. When all real-time change data entries in the retransmitted data subset match, the synchronization master node determines that the data packet is valid as a whole.

[0197] S46. If the number of failed retransmissions exceeds a preset threshold, disconnect from the slave node, wait for the slave node to reconnect, and then send the full real-time database profile data to the slave node for recovery.

[0198] Specifically, when the number of failed micro-window retransmission verifications for the same slave node accumulates to exceed a preset threshold within a preset time window, the master node determines that the link where the slave node resides has a serious quality problem. The master node disconnects the TCP connection with the slave node and generates an alarm message. After detecting a heartbeat timeout or abnormal TCP connection disconnection with the master node, the slave node automatically re-initiates a connection request to the master node. The master node re-performs identity verification and software environment compatibility verification for the successfully reconnected slave node. After passing the verification, the master node sends the current full real-time database profile data, combining the runtime state and application, to the slave node. After receiving and loading the full real-time database profile data, the slave node resumes the incremental synchronization process.

[0199] Please see Figure 6 The present invention also provides a consensus-driven, multi-state, multi-application real-time synchronization system, the system comprising:

[0200] The master-slave node election module 1 is used to elect a unique master node from the candidate node group based on the combination formed by matching a single application with a single running state, and to load the remaining nodes of the same type as slave nodes.

[0201] Data capture module 2 is used to simultaneously write real-time changed data to both the real-time database and the local cache queue when an application on any node generates such data. It also continuously monitors the local cache queue, reading, aggregating, and compressing newly added real-time changed data to generate data packets.

[0202] Consensus verification module 3 is used by slave nodes to generate aggregated verification information for integrity verification from data packets belonging to the same group, and then send the data packets and aggregated verification information together to the master node corresponding to that group. After receiving the data, the master node calculates its local aggregated verification information and compares it. If they match, the data is deemed legal and valid; if they do not match, anomaly recovery processing is triggered.

[0203] Specifically, during data synchronization, this invention abandons the traditional approach of performing complete consensus verification on every data packet, and instead adopts a batch consensus mechanism based on a sliding window. This improves data synchronization efficiency and reduces the impact on system real-time performance. The following is the complete structure and execution flow of the batch consensus verification algorithm:

[0204] 1. Sliding Window Module: A sliding window is set at the slave node. This window is used to define a data set within a certain time range or a certain data volume range. The size of the window can be dynamically adjusted according to the actual application scenario's requirements for the real-time performance and accuracy of data synchronization.

[0205] 2. Data Packaging Module: This module is responsible for collecting and organizing data from the node's sliding window. Specifically, it filters out data with similar properties and applications, and packages this data according to a specific format. During the packaging process, it not only ensures the integrity and order of the data but also compresses the data to reduce bandwidth consumption during subsequent transmission. Simultaneously, it requires encrypted transmission of control data.

[0206] 3. Aggregate Verification Information Generation Module: This module is one of the core components of the batch consensus verification algorithm. It performs Merkle tree calculations on the packaged data to generate a unique aggregate verification information. This aggregate verification information comprehensively reflects the characteristics of all data within the window, similar to a digital fingerprint of the data. By verifying this aggregate verification information, it can be determined whether the data within the window has been tampered with or lost as a whole.

[0207] 4. Data Transmission Module: This module combines the header containing aggregated verification information generated by the slave node with the compressed data packet, and then sends the complete data packet to the master node via a preset communication protocol. During transmission, to ensure data reliability and security, encryption and other methods can be used to prevent data theft or tampering.

[0208] 5. Master Node Verification Module: After receiving a data packet from a slave node, the master node first extracts the aggregated verification information from the packet header. Then, the master node uses the same algorithm as the slave node to calculate the received compressed data packet and generate local aggregated verification information. The locally generated aggregated verification information is then compared and verified with the aggregated verification information sent by the slave node.

[0209] To handle potential anomalies, the system also includes a retry mechanism for verification failures. When the master node detects a mismatch in aggregated verification information, it immediately disconnects the TCP connection with the slave node and generates an alarm. The slave node re-establishes a connection with the master node after the heartbeat message times out. The master node then re-performs consensus verification and sends cross-sectional data to ensure high consistency and accuracy of real-time data from the same application in the same state. The data forwarding process between the slave and master nodes is as follows: Figure 5 As shown.

[0210] The targeted distribution synchronization module 4 is used by the master node to forward legitimate and valid data packets to all other slave nodes in the local area network that have the same combination. After receiving the data packets, each slave node writes the real-time changes in the data packets into the real-time database, thus completing an incremental synchronization.

[0211] In summary, by utilizing the above-mentioned technical solutions of this invention, the present invention achieves fine-grained isolation of the synchronization link by independently electing the master node based on both the runtime state and the application. Different runtime states and different business applications each have independent master nodes and dedicated transmission links. Synchronization anomalies or master node failures in any combination are isolated within that combination and will not affect the synchronization process of other runtime states or other business applications. Simultaneously, the same physical node can act as either a master node or a slave node in different combinations depending on its loaded state and application, achieving flexible reuse and load balancing of node resources and effectively avoiding performance bottlenecks and single-point-of-failure risks under the traditional single-master-node architecture. This invention significantly reduces the interaction overhead of consensus verification through a batch consensus verification mechanism based on a sliding window. Slave nodes aggregate multiple real-time changing data from the same runtime state and the same application into data packets, generating aggregated verification information on a data packet basis. The master node only needs to verify the data packets once to confirm the integrity of the batch data, without needing to confirm each data packet individually. Meanwhile, the transmission links for each operating state and each application are independent of each other, and data is transmitted in parallel on each link without interference, avoiding transmission congestion and latency accumulation caused by data mixing, and effectively improving transmission efficiency in multi-state and multi-application concurrent synchronization scenarios. This invention ensures eventual consistency of data synchronization through a multi-level anomaly recovery mechanism; when verification fails, the master node accurately locates the range of erroneous data according to the micro-window index mapping table and performs incremental compensation, significantly reducing the recovery cost; full-section recovery is only triggered when incremental compensation repeatedly fails, avoiding excessive consumption of system resources by frequent full synchronization. Slave nodes automatically detect connection anomalies through a heartbeat mechanism and initiate reconnection. After reconnection, they automatically receive full-section data and restore to a consistent state, ensuring that data from each node can still quickly recover consistency in abnormal scenarios such as node restarts and network outages, meeting the stringent requirements of power systems for data reliability.

[0212] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

Claims

1. A consensus-driven, multi-application real-time synchronization method, characterized in that, include: S1. Based on the combination formed by matching a single application with a single running state, elect a unique master node from the candidate node group, and load the remaining nodes with the same type of combination as slave nodes. S2. When an application on any node generates real-time changing data, the real-time changing data is simultaneously written to the real-time database and the local cache queue. It continuously monitors the local cache queue, reads, collects, and compresses newly added real-time change data to generate data packets; S3. The slave node generates aggregated verification information for integrity verification for data packets belonging to the same group, and sends the data packets and aggregated verification information together to the master node corresponding to the group; after receiving the data packets, the master node calculates the local aggregated verification information and compares it. If they match, the data is determined to be legal and valid. If they do not match, an exception recovery process is triggered. S4. The master node forwards the valid data packets to all other slave nodes in the local area network that have the same combination. After receiving the data packets, each slave node writes the real-time changes in the data packets into the real-time database, thus completing an incremental synchronization.

2. The consensus-driven, multi-application real-time synchronization method according to claim 1, characterized in that, The process of selecting a unique master node from a candidate node group based on a combination formed by matching a single running state with a single application, and then using the remaining nodes loaded with the same type of combination as slave nodes, includes: S11. Perform a full verification and veto process on each candidate node in the candidate node group. If any verification item does not meet the preset screening threshold, the candidate node is vetoed. S12. If the list of available nodes after filtering is empty, expand the candidate node group and restart the initial screening process; if only one candidate node remains after filtering, determine that candidate node as the main node of the current running state and application combination; if multiple candidate nodes remain after filtering, execute S13. S13. Calculate the cumulative load value of all master node roles that the candidate node has assumed, based on the baseline load weight of each master node role; if the cumulative load value exceeds the preset resource threshold of the candidate node, the candidate node loses its candidate status for this round and is selected from the remaining candidate nodes. S14. For each candidate node that passes the initial screening, obtain the synchronization operation index of the candidate node for the current combination. If the synchronization operation index exceeds the preset index threshold, the candidate node will be vetoed in this round of election. The synchronization operation index includes the backlog depth of the window to be confirmed. S15. Obtain the operating indicators of the remaining candidate nodes after screening, calculate the comprehensive score according to the preset weighting formula, and determine the candidate node with the highest score as the master node of the current running state and application combination; if the comprehensive scores are the same, randomly select any candidate node as the master node.

3. The consensus-driven, multi-application real-time synchronization method according to claim 1, characterized in that, The slave node generates aggregated verification information for integrity verification from data packets belonging to the same group, and sends the data packets and aggregated verification information together to the master node corresponding to that group, including: S31. Based on the sliding window mechanism, according to the preset time window length or data volume window size, multiple real-time changing data belonging to the same running state and application are aggregated into a data packet. S32. Preset differentiated window parameter baselines according to the type of operating state, and dynamically adjust the sliding window size in real time according to the current network operating indicators, and the adjustment range of the sliding window size is constrained within the parameter boundary of the corresponding operating state. S33. The Merkle tree algorithm is used to calculate and generate unique aggregated verification information for the data packet; the aggregated verification information serves as the digital fingerprint of the data packet. S34. The aggregated verification information is used as the data packet header and combined with the compressed data packet to form a complete message. The complete message is sent to the corresponding master node through the dedicated independent transmission control protocol link of the current running state and the application. The data packet header contains the total Merkle root hash and the sub-Merkle root hash, the micro-window index and the semantic tag mapping table.

4. The consensus-driven, multi-application real-time synchronization method according to claim 3, characterized in that, The Merkle tree algorithm is used to calculate and generate unique aggregated verification information for the data packets, including: S331. Extract business semantic features from all real-time change data within the sliding window, and generate a multi-dimensional semantic feature vector and business semantic label corresponding to each piece of real-time change data. S332. Based on the semantic association between the multidimensional semantic feature vector and the business semantic tag, perform clustering with capacity constraints to obtain several semantic clusters; each semantic cluster corresponds to a micro-window, and the real-time changing data within the same semantic cluster has a business logic association; S333. Calculate the submerkel root hash for all real-time changing data in each micro-window; S334. Generate the total Merkle root hash of the next level as the aggregated verification information by using all the child Merkle root hashes as leaf nodes.

5. The consensus-driven, multi-application real-time synchronization method according to claim 4, characterized in that, Based on the semantic association between the multidimensional semantic feature vectors and business semantic tags, a clustering process with capacity constraints is performed to obtain several semantic clusters, including: S3321. The optimization objective is to maximize the mutual information between the micro-window partitioning result and the data business semantic tags, and the preset lower and upper limits of the number of data entries in the micro-window are used as constraints. S3322. Initialize each micro-window as an empty cluster, randomly select a real-time changing data as the seed data of the first micro-window, and then distribute the remaining real-time changing data to the nearest micro-window according to the current semantic similarity to generate an initial partitioning scheme. S3323. Calculate the mutual information between each micro-window and the corresponding data service semantic tag under the current partitioning scheme. If the mutual information is lower than the preset threshold, perform the micro-window splitting operation. S3324. Iteratively execute data migration attempts between micro-windows, recalculate mutual information after each migration. If the mutual information increases after migration and does not violate capacity constraints, retain the migration result; if the mutual information no longer increases or reaches the maximum number of iterations in multiple consecutive iterations, terminate the iteration. S3325. The partitioning result with the largest mutual information at the end of the iteration is taken as the micro-window partitioning result.

6. The consensus-driven, multi-application real-time synchronization method according to claim 5, characterized in that, The microwindow splitting operation includes: S33231. Calculate the marginal contribution value of each real-time changing data in the current micro-window to the mutual information content; the marginal contribution value is the decrease in the mutual information content between the micro-window and the business semantic tag after removing the data; S33232: Mark data whose marginal contribution value is lower than the preset splitting threshold as splittable data, and remove the splittable data from the current micro-window to form a new candidate micro-window; S33233. Check whether the number of real-time change data entries in the newly formed candidate micro-window is lower than the preset lower limit. If it is lower than the lower limit, cancel the splitting operation and redistribute the splittable data to the adjacent micro-window with the least mutual information loss.

7. The consensus-driven, multi-application real-time synchronization method according to claim 3, characterized in that, The step of using the aggregated verification information as a data packet header, combining it with the compressed data packet to form a complete message, and sending the complete message to the corresponding master node through the dedicated independent transmission control protocol link of the current running state and application includes: S341. The slave node combines the currently combined complete message with the dependency sequence number snapshot array carried in the cached data structure, encapsulates it in the extended header of the complete message, and sends it to the master node. S342. After the master node receives the complete message and completes consensus verification, it parses the dependency sequence number snapshot array in the extended packet header and compares each dependency sequence number in the dependency sequence number snapshot array with the latest globally confirmed sequence number of the corresponding running state and the corresponding application recorded on the local machine. S343. If any dependent sequence number is found to be lagging behind the current global progress by more than a preset threshold, a dependency barrier instruction is appended to the extended field of the Transmission Control Protocol acknowledgment response message returned to the slave node; the dependency barrier instruction includes the lagging dependent runtime identifier, the dependent application identifier, and the target global sequence number to be filled in. S344. After receiving the Transmission Control Protocol confirmation response with the Dependency Fence instruction from the slave node, the incremental synchronization process of the current combination is paused. The slave node first requests the missing real-time change data between the current local sequence number and the target global sequence number from the master node. After the missing data is filled in, the incremental synchronization process of the current sliding window is resumed.

8. The consensus-driven, multi-application real-time synchronization method according to claim 1, characterized in that, The master node forwards valid data packets to all other slave nodes within the local area network that load the same combination. Each slave node, upon receiving the packet, writes the real-time changes in the data packet into the real-time database, completing one incremental synchronization cycle, which includes: S41. After receiving the complete message, the master node extracts the aggregated verification information from the data packet header of the complete message and parses the compressed data packet in the complete message. S42. The master node uses the same Merkle tree algorithm as the slave node to calculate local aggregate check information for the decompressed data packets. S43. Compare the local aggregated verification information with the aggregated verification information sent by the slave node; S44. If the total Merkle root hash matches, the data packet is determined to be complete and tamper-proof, and receipt is confirmed. S45. If the comparison is inconsistent, the master node locates the target micro-window with a hash mismatch based on the data packet header, requests the slave node to resend the data subset corresponding to the target micro-window, and performs a comparison item by item; if the resent data verification passes, the data packet is determined to be valid as a whole. S46. If the number of failed retransmissions exceeds a preset threshold, disconnect from the slave node, wait for the slave node to reconnect, and then send the full real-time database profile data to the slave node for recovery.

9. The consensus-driven, multi-application real-time synchronization method according to claim 8, characterized in that, The master node locates the target micro-window with hash mismatch based on the data packet header, requests the slave node to resend the data subset corresponding to the target micro-window, and performs a comparison item by item. If the retransmitted data verification passes, the data packet is considered valid overall, including: S451. The master node compares the submerkle root hash of each micro-window with the submerkle root hash, micro-window index and semantic tag mapping table in the data packet header one by one, and locates the target micro-window index of all hash mismatches. S452. The master node sends a location request back to the slave node. The location request includes a list of target micro-window indexes. After receiving the location request, the slave node repackages the original data subset corresponding to the target micro-window indexes and sends it to the master node. S453. The master node compares each retransmitted data subset to locate the erroneous data entry; if the retransmitted data passes the verification, the data packet is deemed valid as a whole.

10. A consensus-driven multi-state multi-application real-time synchronization system, used to implement the consensus-driven multi-state multi-application real-time synchronization method according to any one of claims 1-9, characterized in that, The system includes: The master-slave node election module is used to elect a unique master node from the candidate node group based on the combination formed by matching a single application with a single running state, and to load the remaining nodes with the same type of combination as slave nodes. The data capture module is used to write real-time changed data to both the real-time database and the local cache queue when the application on any node generates real-time changed data; and to continuously monitor the local cache queue, read, collect and compress the newly added real-time changed data, and generate data packets. The consensus verification module is used by the slave node to generate aggregate verification information for integrity verification from data packets belonging to the same group, and send the data packets and aggregate verification information together to the master node corresponding to the group; after receiving the data, the master node calculates the local aggregate verification information and compares it. If they match, the data is determined to be legal and valid. If they do not match, an exception recovery process is triggered. The targeted distribution synchronization module is used by the master node to forward legitimate and valid data packets to all other slave nodes in the local area network that have the same combination of data. After receiving the data packets, each slave node writes the real-time changes in the data packets into the real-time database, thus completing an incremental synchronization.

Images