A multi-node event synchronization device

Abstract

This invention relates to a multi-node event synchronization device, belonging to the emerging information technology terminal field. It includes the following steps: an MDN node, an EN node, a PackBatchWorker component, a ReplicationWorker component, and an SDN node. This invention proposes a queue-based incremental update method and a full hash value comparison method. Events are enqueued sequentially, ensuring event order consistency, while full hash value comparison ensures the consistency of all data. Furthermore, a three-level queue is proposed. The first-level queue handles event writing operations, enabling non-blocking writing and improving the event writing rate. The second-level queue is mainly responsible for event packaging and merging; merging reduces the frequency of event synchronization without affecting the first-level queue's writing. The third-level queue is mainly responsible for event synchronization operations between nodes, without affecting event writing and packaging. All operations are memory-based, thus exhibiting high performance.

Description

Technical Field

[0001] This invention belongs to the field of software development technology, and in particular relates to a multi-node event synchronization device. Background Technology

[0002] In distributed systems, information synchronization between nodes is crucial to ensure system consistency and reliability. Below are some common inter-node information synchronization mechanisms:

[0003] Message passing: Nodes synchronize information through message passing. Nodes can notify other nodes of state changes or events by sending messages. Message passing can be synchronous (e.g., request-response pattern) or asynchronous (e.g., publish-subscribe pattern), depending on the system requirements and design.

[0004] Broadcast: Broadcast is a synchronization mechanism that transmits information to all nodes. When a node undergoes a state change or an event occurs, it can broadcast the message to all other nodes. Broadcast can be implemented through multicast or multi-cast mechanisms to improve efficiency and reduce network bandwidth usage.

[0005] Shared log: A shared log is a data structure that records node operations and state changes. All nodes share the same log. When a node's state changes, it appends the corresponding entry to the log and copies it to other nodes. Through the shared log, nodes can perform the same operations in the same order to maintain consistency.

[0006] In the aforementioned synchronization mechanisms, due to the asynchronicity of distributed systems and network latency, data synchronization between nodes may be delayed or inconsistent. The order and timestamps of data updates may lead to inconsistencies between different nodes due to network communication delays or failures. Furthermore, in large-scale distributed systems with numerous nodes and massive amounts of data, the performance of information synchronization becomes a critical issue. Efficiently propagating data updates to all nodes and handling a large number of synchronization requests requires consideration of system performance and scalability. Summary of the Invention

[0007] In view of the shortcomings of the prior art, the purpose of the invention is to provide a multi-node event synchronization device, which includes a distributed caching system, where MDN nodes receive cache update requests and synchronize update operations to other SDN cache nodes to ensure data consistency and high availability; a distributed log system, where MDN nodes are responsible for receiving and writing log messages and then synchronizing the log messages to other SDN nodes to achieve centralized storage and distribution of logs; and a distributed message queue system, where MDN nodes are responsible for receiving and processing messages and synchronizing the messages to other SDN nodes to achieve reliable message delivery and consumption.

[0008] This invention proposes a multi-node event synchronization device, including an MDN node, an EN node, a PackBatchWorker component, a ReplicationWorker component, and an SDN node;

[0009] The EN node is used to push events to the MDN node;

[0010] The MDN node is used to process the received events through the Dispatcher component;

[0011] The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events;

[0012] The ReplicationWorker component is used to send BatchEvent to the SDN node;

[0013] The SDN node is used to receive the BatchEvent synchronized by the MDN node, break down the BatchEvent, process the Event through the Dispatcher, update its own EventResultBucket, perform hash calculation on its own EventResultBucket to generate a hash value based on the premise that the current BatchEvent is the last batch event, compare it with the hash value obtained from synchronization, and mark its own status.

[0014] Furthermore, the MDN node is a coordinating node, and the MDN node is used for writing events and synchronizing events.

[0015] Furthermore, the MDN node is used to process the received events through the Dispatcher component, specifically including the following steps:

[0016] S1: Receive events pushed by the EN node through the MDN node;

[0017] S2: The Dispatcher processes the events received by the MDN node, puts the corresponding event results into the EventResultBucket, and puts the events into the AcceptorQueue.

[0018] Furthermore, the PackBatchWorker component, used to transfer events, generate BatchEvents, and mark a last batch of events, specifically includes the following steps:

[0019] S3: The PackBatchWorker component transfers events from the AcceptorQueue to the ProcessingQueue at preset intervals;

[0020] S4: Then take the first Event from the ProcessingQueue pair, and sequentially traverse the queue until the second Event is found, and the enqueue time of the Event is greater than or equal to the preset time;

[0021] S5: Package the events between the two events to generate a BatchEvent and put it into the BatchEventQueue. If the current BatchEvent is the last batch event in this operation, perform a hash calculation on the EventResultBucket in the current MDN node, put the calculated hash value into the BatchEvent, and mark the BatchEvent as the last batch event.

[0022] Furthermore, the ReplicationWorker component for sending BatchEvents to SDN nodes specifically includes the following steps:

[0023] S6: The ReplicationWorker component retrieves elements from the BatchEventQueue at preset intervals, iterates through PeerNodes, and sends the BatchEvent to other SDN nodes.

[0024] Furthermore, the SDN node is used to receive BatchEvents synchronized from the MDN node, break down the BatchEvents, process the Events through the Dispatcher, update its own EventResultBucket, and, assuming that the current BatchEvent is the last batch of events, calculate a hash value for its own EventResultBucket using a hash algorithm, compare it with the hash value obtained from synchronization, and mark its own status. The specific steps include:

[0025] S7: Receive BatchEvent synchronized from MDN node through SDN node and break down BatchEvent;

[0026] S8: Process different Events through the Dispatcher, and then update its own EventResultBucket. If the current BatchEvent is found to be the last batch of events, perform a hash calculation on its own EventResultBucket and compare the obtained hash value with the synchronized hash value.

[0027] S9: If the two hash values ​​are consistent, mark its own state as complete; if the two hash values ​​are inconsistent, mark its own state as invalid, actively pull the entire EventResultBucket from the MDN node, and mark it as complete.

[0028] Furthermore, in S3 and S4, the preset time is 500ms.

[0029] Furthermore, in S5, a BatchEvent can contain a maximum of 250 Events.

[0030] Furthermore, in S6, the preset time is 30 seconds.

[0031] Furthermore, the hash algorithm includes the MD5 algorithm, the CRC algorithm, and the MurmurHash algorithm.

[0032] The beneficial effects of this invention are as follows:

[0033] 1. This invention proposes a method based on queue-based incremental update and full hash value comparison. Events are enqueued in order to ensure the consistency of event order, while full hash value comparison ensures the consistency of all data. Hash comparison is a relatively heavyweight operation with wide applications in distributed systems.

[0034] 2. This invention proposes a three-level queue. The first-level queue is responsible for writing events, which can achieve non-blocking writing and thus improve the event writing rate. The second-level queue is mainly responsible for packing and merging events. Without affecting the writing of the first-level queue, the frequency of event synchronization is reduced by merging. The third-level queue is mainly responsible for the synchronization operation of events between nodes. It does not affect the writing and packing of events. At the same time, all operations are memory operations, so it has high performance.

[0035] 3. This invention allows for dynamic setting of the polling time of PackBatchWorker and ReplicationWorker, enabling different event intervals to be set for different business scenarios, thereby improving timeliness.

[0036] 4. This invention sets up a data recovery process. When MDN receives an event, it first writes the log to disk and then processes the event. When the service restarts, it recovers the data by replaying the log, thus ensuring that the data is not lost. When writing the log to disk, it uses MappedFile technology. Through the file memory mapping mechanism, the log data is first written to memory and then flushed from memory to disk, thus having a high write speed. Attached Figure Description

[0037] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. It is obvious that the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.

[0038] Figure 1 This is a flowchart of an event processing procedure based on a three-level queue, according to an embodiment of the present invention.

[0039] Figure 2 This is a flowchart illustrating the incremental update and full hash value comparison processing of an embodiment of the present invention.

[0040] Figure 3 This is a schematic diagram of a three-level queue according to an embodiment of the present invention;

[0041] Figure 4 This is a flowchart illustrating log synchronization and disk persistence in an embodiment of the present invention. Detailed Implementation

[0042] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0043] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.

[0044] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the invention as detailed in the appended claims.

[0045] This invention proposes a multi-node event synchronization device to solve the network request storm caused by synchronizing a large number of messages, as well as the problems of data consistency and performance.

[0046] To aid in understanding this invention, the following explanations are provided for the terms used herein:

[0047] EN (Event Node) is the event-triggered node that sends the event to MDN;

[0048] The MDN (Master Data Node) is responsible for writing events and synchronizing events with other SDNs.

[0049] SDN (SLAVE Data Node) receives events synchronized from MDN nodes and provides read operations for these events.

[0050] Event, various types of messages and events;

[0051] BatchEvent is a collection of multiple events.

[0052] PeerNodes are data structures used in MDN to store data from other SDN nodes.

[0053] EventResultBucket is a data structure used in MDN and SDN nodes to store the results of various event processing.

[0054] Dispatcher, a component in MDN and SDN used to handle various events;

[0055] AcceptorQueue, a queue in MDN used to receive various events;

[0056] ProcessingQueue, a queue in MDN used for secondary processing of events;

[0057] BatchEventQueue is a queue in MDN nodes used to store batch events.

[0058] PackBatchWorker is a component in MDN nodes used to transfer events from the AcceptorQueue to the ProcessingQueue, package the Events into BatchEvents, and transfer them to the BatchEventQueue.

[0059] ReplicationWorker is a component in MDN used to synchronize BatchEvents in the BatchEventQueue to other SDNs.

[0060] Furthermore, this invention proposes a multi-node event synchronization device, which, for the purpose of explanation, is as follows: Figures 1-4 As shown, the following implementation method is provided:

[0061] Implementation Method 1

[0062] A multi-node event synchronization device, characterized in that it includes MDN nodes, EN nodes, PackBatchWorker components, ReplicationWorker components, and SDN nodes.

[0063] The EN node is used to push events to the MDN node;

[0064] The MDN node is used to process the received events through the Dispatcher component;

[0065] The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events;

[0066] The ReplicationWorker component is used to send BatchEvent to the SDN node;

[0067] The SDN node is used to receive the BatchEvent synchronized by the MDN node, break down the BatchEvent, process the Event through the Dispatcher, update its own EventResultBucket, perform hash calculation on its own EventResultBucket to generate a hash value based on the premise that the current BatchEvent is the last batch event, compare it with the hash value obtained from synchronization, and mark its own status.

[0068] Implementation Method 2

[0069] A multi-node event synchronization device, characterized in that it includes MDN nodes, EN nodes, PackBatchWorker components, ReplicationWorker components, and SDN nodes.

[0070] The EN node is used to push events to the MDN node;

[0071] The MDN node is used to process the received events through the Dispatcher component;

[0072] The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events;

[0073] The ReplicationWorker component is used to send BatchEvent to the SDN node;

[0074] The SDN node is used to receive the BatchEvent synchronized by the MDN node, break down the BatchEvent, process the Event through the Dispatcher, update its own EventResultBucket, perform hash calculation on its own EventResultBucket to generate a hash value based on the premise that the current BatchEvent is the last batch event, compare it with the hash value obtained from synchronization, and mark its own status.

[0075] The MDN node is a coordinating node, and it is used for writing events and synchronizing events.

[0076] Implementation Method 3

[0077] A multi-node event synchronization device, characterized in that it includes MDN nodes, EN nodes, PackBatchWorker components, ReplicationWorker components, and SDN nodes.

[0078] The EN node is used to push events to the MDN node;

[0079] The MDN node is used to process the received events through the Dispatcher component;

[0080] The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events;

[0081] The ReplicationWorker component is used to send BatchEvent to the SDN node;

[0082] The SDN node is used to receive the BatchEvent synchronized by the MDN node, break down the BatchEvent, process the Event through the Dispatcher, update its own EventResultBucket, perform hash calculation on its own EventResultBucket to generate a hash value based on the premise that the current BatchEvent is the last batch event, compare it with the hash value obtained from synchronization, and mark its own status.

[0083] The MDN node is a coordinating node, and it is used for writing events and synchronizing events.

[0084] The MDN node is used to process received events through the Dispatcher component, specifically including the following steps:

[0085] S1: Receive events pushed by the EN node through the MDN node;

[0086] S2: The Dispatcher processes the events received by the MDN node, puts the corresponding event results into the EventResultBucket, and puts the events into the AcceptorQueue.

[0087] Implementation Method 4

[0088] A multi-node event synchronization device, characterized in that it includes MDN nodes, EN nodes, PackBatchWorker components, ReplicationWorker components, and SDN nodes.

[0089] The EN node is used to push events to the MDN node;

[0090] The MDN node is used to process the received events through the Dispatcher component;

[0091] The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events;

[0092] The ReplicationWorker component is used to send BatchEvent to the SDN node;

[0093] The SDN node is used to receive the BatchEvent synchronized by the MDN node, break down the BatchEvent, process the Event through the Dispatcher, update its own EventResultBucket, perform hash calculation on its own EventResultBucket to generate a hash value based on the premise that the current BatchEvent is the last batch event, compare it with the hash value obtained from synchronization, and mark its own status.

[0094] The MDN node is a coordinating node, and it is used for writing events and synchronizing events.

[0095] The MDN node is used to process received events through the Dispatcher component, specifically including the following steps:

[0096] S1: Receive events pushed by the EN node through the MDN node;

[0097] S2: The Dispatcher processes the events received by the MDN node, puts the corresponding event results into the EventResultBucket, and puts the events into the AcceptorQueue.

[0098] The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events. Specifically, it includes the following steps:

[0099] S3: The PackBatchWorker component transfers events from the AcceptorQueue to the ProcessingQueue at preset intervals;

[0100] S4: Then take the first Event from the ProcessingQueue pair, and sequentially traverse the queue until the second Event is found, and the enqueue time of the Event is greater than or equal to the preset time;

[0101] S5: Package the events between the two events to generate a BatchEvent and put it into the BatchEventQueue. If the current BatchEvent is the last batch event in this operation, perform a hash calculation on the EventResultBucket in the current MDN node, put the calculated hash value into the BatchEvent, and mark the BatchEvent as the last batch event.

[0102] Implementation Method 5

[0103] A multi-node event synchronization device, characterized in that it includes MDN nodes, EN nodes, PackBatchWorker components, ReplicationWorker components, and SDN nodes.

[0104] The EN node is used to push events to the MDN node;

[0105] The MDN node is used to process the received events through the Dispatcher component;

[0106] The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events;

[0107] The ReplicationWorker component is used to send BatchEvent to the SDN node;

[0108] The SDN node is used to receive the BatchEvent synchronized by the MDN node, break down the BatchEvent, process the Event through the Dispatcher, update its own EventResultBucket, perform hash calculation on its own EventResultBucket to generate a hash value based on the premise that the current BatchEvent is the last batch event, compare it with the hash value obtained from synchronization, and mark its own status.

[0109] The MDN node is a coordinating node, and it is used for writing events and synchronizing events.

[0110] The MDN node is used to process received events through the Dispatcher component, specifically including the following steps:

[0111] S1: Receive events pushed by the EN node through the MDN node;

[0112] S2: The Dispatcher processes the events received by the MDN node, puts the corresponding event results into the EventResultBucket, and puts the events into the AcceptorQueue.

[0113] The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events. Specifically, it includes the following steps:

[0114] S3: The PackBatchWorker component transfers events from the AcceptorQueue to the ProcessingQueue at preset intervals;

[0115] S4: Then take the first Event from the ProcessingQueue pair, and sequentially traverse the queue until the second Event is found, and the enqueue time of the Event is greater than or equal to the preset time;

[0116] S5: Package the events between the two events to generate a BatchEvent and put it into the BatchEventQueue. If the current BatchEvent is the last batch event in this operation, perform a hash calculation on the EventResultBucket in the current MDN node, put the calculated hash value into the BatchEvent, and mark the BatchEvent as the last batch event.

[0117] The ReplicationWorker component is used to send BatchEvents to SDN nodes, specifically including the following steps:

[0118] S6: The ReplicationWorker component retrieves elements from the BatchEventQueue at preset intervals, iterates through PeerNodes, and sends the BatchEvent to other SDN nodes.

[0119] Implementation Method 6

[0120] A multi-node event synchronization device, characterized in that it includes MDN nodes, EN nodes, PackBatchWorker components, ReplicationWorker components, and SDN nodes.

[0121] The EN node is used to push events to the MDN node;

[0122] The MDN node is used to process the received events through the Dispatcher component;

[0123] The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events;

[0124] The ReplicationWorker component is used to send BatchEvent to the SDN node;

[0125] The SDN node is used to receive the BatchEvent synchronized by the MDN node, break down the BatchEvent, process the Event through the Dispatcher, update its own EventResultBucket, perform hash calculation on its own EventResultBucket to generate a hash value based on the premise that the current BatchEvent is the last batch event, compare it with the hash value obtained from synchronization, and mark its own status.

[0126] The MDN node is a coordinating node, and it is used for writing events and synchronizing events.

[0127] The MDN node is used to process received events through the Dispatcher component, specifically including the following steps:

[0128] S1: Receive events pushed by the EN node through the MDN node;

[0129] S2: The Dispatcher processes the events received by the MDN node, puts the corresponding event results into the EventResultBucket, and puts the events into the AcceptorQueue.

[0130] The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events. Specifically, it includes the following steps:

[0131] S3: The PackBatchWorker component transfers events from the AcceptorQueue to the ProcessingQueue at preset intervals;

[0132] S4: Then take the first Event from the ProcessingQueue pair, and sequentially traverse the queue until the second Event is found, and the enqueue time of the Event is greater than or equal to the preset time;

[0133] S5: Package the events between the two events to generate a BatchEvent and put it into the BatchEventQueue. If the current BatchEvent is the last batch event in this operation, perform a hash calculation on the EventResultBucket in the current MDN node, put the calculated hash value into the BatchEvent, and mark the BatchEvent as the last batch event.

[0134] The ReplicationWorker component is used to send BatchEvents to SDN nodes, specifically including the following steps:

[0135] S6: The ReplicationWorker component retrieves elements from the BatchEventQueue at preset intervals, iterates through PeerNodes, and sends the BatchEvent to other SDN nodes.

[0136] The SDN node is used to receive BatchEvents synchronized from the MDN node, break down the BatchEvents, process the Events through the Dispatcher, update its own EventResultBucket, and, assuming that the current BatchEvent is the last batch of events, calculate a hash value for its own EventResultBucket using a hash algorithm, compare it with the hash value obtained from synchronization, and mark its own status. The specific steps include:

[0137] S7: Receive BatchEvent synchronized from MDN node through SDN node and break down BatchEvent;

[0138] S8: Process different Events through the Dispatcher, and then update its own EventResultBucket. If the current BatchEvent is found to be the last batch of events, perform a hash calculation on its own EventResultBucket and compare the obtained hash value with the synchronized hash value.

[0139] S9: If the two hash values ​​are consistent, mark its own state as complete; if the two hash values ​​are inconsistent, mark its own state as invalid, actively pull the entire EventResultBucket from the MDN node, and mark it as complete.

[0140] Implementation Method 7

[0141] A multi-node event synchronization device, characterized in that it includes MDN nodes, EN nodes, PackBatchWorker components, ReplicationWorker components, and SDN nodes.

[0142] The EN node is used to push events to the MDN node;

[0143] The MDN node is used to process the received events through the Dispatcher component;

[0144] The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events;

[0145] The ReplicationWorker component is used to send BatchEvent to the SDN node;

[0146] The SDN node is used to receive the BatchEvent synchronized by the MDN node, break down the BatchEvent, process the Event through the Dispatcher, update its own EventResultBucket, perform hash calculation on its own EventResultBucket to generate a hash value based on the premise that the current BatchEvent is the last batch event, compare it with the hash value obtained from synchronization, and mark its own status.

[0147] The MDN node is a coordinating node, and it is used for writing events and synchronizing events.

[0148] The MDN node is used to process received events through the Dispatcher component, specifically including the following steps:

[0149] S1: Receive events pushed by the EN node through the MDN node;

[0150] S2: The Dispatcher processes the events received by the MDN node, puts the corresponding event results into the EventResultBucket, and puts the events into the AcceptorQueue.

[0151] The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events. Specifically, it includes the following steps:

[0152] S3: The PackBatchWorker component transfers events from the AcceptorQueue to the ProcessingQueue at preset intervals;

[0153] S4: Then take the first Event from the ProcessingQueue pair, and sequentially traverse the queue until the second Event is found, and the enqueue time of the Event is greater than or equal to the preset time;

[0154] S5: Package the events between the two events to generate a BatchEvent and put it into the BatchEventQueue. If the current BatchEvent is the last batch event in this operation, perform a hash calculation on the EventResultBucket in the current MDN node, put the calculated hash value into the BatchEvent, and mark the BatchEvent as the last batch event.

[0155] In both S3 and S4, the preset time is 500ms.

[0156] Implementation Method 8

[0157] A multi-node event synchronization device, characterized in that it includes MDN nodes, EN nodes, PackBatchWorker components, ReplicationWorker components, and SDN nodes.

[0158] The EN node is used to push events to the MDN node;

[0159] The MDN node is used to process the received events through the Dispatcher component;

[0160] The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events;

[0161] The ReplicationWorker component is used to send BatchEvent to the SDN node;

[0162] The SDN node is used to receive the BatchEvent synchronized by the MDN node, break down the BatchEvent, process the Event through the Dispatcher, update its own EventResultBucket, perform hash calculation on its own EventResultBucket to generate a hash value based on the premise that the current BatchEvent is the last batch event, compare it with the hash value obtained from synchronization, and mark its own status.

[0163] The MDN node is a coordinating node, and it is used for writing events and synchronizing events.

[0164] The MDN node is used to process received events through the Dispatcher component, specifically including the following steps:

[0165] S1: Receive events pushed by the EN node through the MDN node;

[0166] S2: The Dispatcher processes the events received by the MDN node, puts the corresponding event results into the EventResultBucket, and puts the events into the AcceptorQueue.

[0167] The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events. Specifically, it includes the following steps:

[0168] S3: The PackBatchWorker component transfers events from the AcceptorQueue to the ProcessingQueue at preset intervals;

[0169] S4: Then take the first Event from the ProcessingQueue pair, and sequentially traverse the queue until the second Event is found, and the enqueue time of the Event is greater than or equal to the preset time;

[0170] S5: Package the events between the two events to generate a BatchEvent and put it into the BatchEventQueue. If the current BatchEvent is the last batch event in this operation, perform a hash calculation on the EventResultBucket in the current MDN node, put the calculated hash value into the BatchEvent, and mark the BatchEvent as the last batch event.

[0171] In S5, a BatchEvent can contain a maximum of 250 Events.

[0172] Implementation Method 9

[0173] A multi-node event synchronization device, characterized in that it includes MDN nodes, EN nodes, PackBatchWorker components, ReplicationWorker components, and SDN nodes.

[0174] The EN node is used to push events to the MDN node;

[0175] The MDN node is used to process the received events through the Dispatcher component;

[0176] The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events;

[0177] The ReplicationWorker component is used to send BatchEvent to the SDN node;

[0178] The SDN node is used to receive the BatchEvent synchronized by the MDN node, break down the BatchEvent, process the Event through the Dispatcher, update its own EventResultBucket, perform hash calculation on its own EventResultBucket to generate a hash value based on the premise that the current BatchEvent is the last batch event, compare it with the hash value obtained from synchronization, and mark its own status.

[0179] The MDN node is a coordinating node, and it is used for writing events and synchronizing events.

[0180] The MDN node is used to process received events through the Dispatcher component, specifically including the following steps:

[0181] S1: Receive events pushed by the EN node through the MDN node;

[0182] S2: The Dispatcher processes the events received by the MDN node, puts the corresponding event results into the EventResultBucket, and puts the events into the AcceptorQueue.

[0183] The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events. Specifically, it includes the following steps:

[0184] S3: The PackBatchWorker component transfers events from the AcceptorQueue to the ProcessingQueue at preset intervals;

[0185] S4: Then take the first Event from the ProcessingQueue pair, and sequentially traverse the queue until the second Event is found, and the enqueue time of the Event is greater than or equal to the preset time;

[0186] S5: Package the events between the two events to generate a BatchEvent and put it into the BatchEventQueue. If the current BatchEvent is the last batch event in this operation, perform a hash calculation on the EventResultBucket in the current MDN node, put the calculated hash value into the BatchEvent, and mark the BatchEvent as the last batch event.

[0187] The ReplicationWorker component is used to send BatchEvents to SDN nodes, specifically including the following steps:

[0188] S6: The ReplicationWorker component retrieves elements from the BatchEventQueue at preset intervals, iterates through PeerNodes, and sends the BatchEvent to other SDN nodes.

[0189] In step S6, the preset time is 30 seconds.

[0190] Implementation Method 10

[0191] A multi-node event synchronization device, characterized in that it includes MDN nodes, EN nodes, PackBatchWorker components, ReplicationWorker components, and SDN nodes.

[0192] The EN node is used to push events to the MDN node;

[0193] The MDN node is used to process the received events through the Dispatcher component;

[0194] The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events;

[0195] The ReplicationWorker component is used to send BatchEvent to the SDN node;

[0196] The SDN node is used to receive the BatchEvent synchronized by the MDN node, break down the BatchEvent, process the Event through the Dispatcher, update its own EventResultBucket, perform hash calculation on its own EventResultBucket to generate a hash value based on the premise that the current BatchEvent is the last batch event, compare it with the hash value obtained from synchronization, and mark its own status.

[0197] The hash algorithms include MD5, CRC, and MurmurHash.

[0198] In summary, to further explain the present invention, another embodiment is provided:

[0199] Implementation Method 11

[0200] MDN is the coordinating node, responsible for writing and synchronizing events.

[0201] EN pushes various events to MDN. After receiving the events, MDN processes them using the Dispatcher, puts the corresponding event results into the EventResultBucket, and puts the events into the AcceptorQueue.

[0202] The PackBatchWorker component transfers events from the AcceptorQueue to the ProcessingQueue every 500ms (polling time can be dynamically configured). It then retrieves the first Event from the ProcessingQueue and sequentially traverses the queue until it finds the second Event, whose enqueue time is at least 500ms longer than the first Event's enqueue time. These two Events are then bundled into a BatchEvent, containing a maximum of 250 Events, and placed into the BatchEventQueue. If the current BatchEvent is the last batch of events in this operation, the EventResultBucket in the current MDN is hashed, and the calculated hash value is added to the BatchEvent, marking it as the last batch of events.

[0203] Every 30 seconds (the polling time can be dynamically configured), ReplicationWorker retrieves elements from BatchEventQueue, iterates through PeerNodes, and sends the BatchEvent to other SDN nodes.

[0204] The SDN node receives the BatchEvent synchronized from the MDN node, breaks down the BatchEvent, processes the different Events using the Dispatcher, and then updates its own EventResultBucket. If the current BatchEvent is found to be the last batch of events, it hashes its own EventResultBucket and compares the resulting hash value with the synchronized hash value. If they match, the node marks its state as complete; otherwise, it marks its state as invalid and actively retrieves the full EventResultBucket from the MDN node, marking it as complete.

[0205] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0206] Memory, used to store computer programs;

[0207] The processor, when executing a program stored in memory, implements the multi-node event synchronization device of the present invention.

[0208] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned terminal and other devices. The memory can include Random Access Memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage system located remotely from the aforementioned processor.

[0209] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0210] Furthermore, to achieve the above objectives, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multi-node event synchronization device of the present invention.

[0211] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable vehicles (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0212] The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0213] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0214] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0215] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "And / or" indicates that either one or both can be chosen. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0216] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-node event synchronization apparatus, characterized by comprising: This includes MDN nodes, EN nodes, PackBatchWorker components, ReplicationWorker components, and SDN nodes. The EN node is used to push events to the MDN node; The MDN node is used to process the received events through the Dispatcher component; The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events; The ReplicationWorker component is used to send BatchEvent to the SDN node; The SDN node is used to receive the BatchEvent synchronized by the MDN node, break down the BatchEvent, process the Event through the Dispatcher, update its own EventResultBucket, perform hash calculation on its own EventResultBucket to generate a hash value based on the premise that the current BatchEvent is the last batch event, compare it with the hash value obtained from synchronization, and mark its own status.

2. The apparatus of claim 1, wherein, The MDN node is a coordinating node, and it is used for writing events and synchronizing events.

3. The apparatus of claim 2, wherein, The MDN node is used to process received events through the Dispatcher component, specifically including the following steps: S1: Receive events pushed by the EN node through the MDN node; S2: The Dispatcher processes the events received by the MDN node, puts the corresponding event results into the EventResultBucket, and puts the events into the AcceptorQueue.

4. The multi-node event synchronization apparatus according to claim 3, wherein The PackBatchWorker component is used to transfer events, generate BatchEvents, and mark a last batch of events. Specifically, it includes the following steps: S3: The PackBatchWorker component transfers events from the AcceptorQueue to the ProcessingQueue at preset intervals; S4: Then take the first Event from the ProcessingQueue pair, and sequentially traverse the queue until the second Event is found, and the enqueue time of the Event is greater than or equal to the preset time; S5: Package the events between the two events to generate a BatchEvent and put it into the BatchEventQueue. If the current BatchEvent is the last batch event in this operation, perform a hash calculation on the EventResultBucket in the current MDN node, put the calculated hash value into the BatchEvent, and mark the BatchEvent as the last batch event.

5. The multi-node event synchronization device according to claim 4, characterized in that, The ReplicationWorker component is used to send BatchEvents to SDN nodes, specifically including the following steps: S6: The ReplicationWorker component retrieves elements from the BatchEventQueue at preset intervals, iterates through PeerNodes, and sends the BatchEvent to other SDN nodes.

6. The multi-node event synchronization device according to claim 5, characterized in that, The SDN node is used to receive BatchEvents synchronized from the MDN node, break down the BatchEvents, process the Events through the Dispatcher, update its own EventResultBucket, and, assuming that the current BatchEvent is the last batch of events, calculate a hash value for its own EventResultBucket using a hash algorithm, compare it with the hash value obtained from synchronization, and mark its own status. The specific steps include: S7: Receive BatchEvent synchronized from MDN node through SDN node and break down BatchEvent; S8: Process different Events through the Dispatcher, and then update its own EventResultBucket. If the current BatchEvent is found to be the last batch of events, perform a hash calculation on its own EventResultBucket and compare the obtained hash value with the synchronized hash value. S9: If the two hash values ​​are consistent, mark its own state as complete; if the two hash values ​​are inconsistent, mark its own state as invalid, actively pull the entire EventResultBucket from the MDN node, and mark it as complete.

7. The multi-node event synchronization device according to claim 4, characterized in that, In both S3 and S4, the preset time is 500ms.

8. A multi-node event synchronization device according to claim 4, characterized in that, In S5, a BatchEvent can contain a maximum of 250 Events.

9. A multi-node event synchronization device according to claim 5, characterized in that, In step S6, the preset time is 30 seconds.

10. A multi-node event synchronization device according to claim 1, characterized in that, The hash algorithms include MD5, CRC, and MurmurHash.

Images