Data synchronization method for decentralized node and network system thereof

Abstract

A data synchronization method for a decentralized node and a network system thereof, which method and system belong to the technical field of communications or networks, and solve the problem of data synchronization of a decentralized node. The network system comprises at least one online network node N5, wherein each network node comprises data D, a request end C and a response end S, and the network node N5 performs data synchronization in the following manner: a request end C5 of the network node N5 sending a request, wherein the request comprises data D5 of the network node N5 itself; after receiving the request from the network node N5 itself or a request from another network node and verifying data in the request, a response end S5 of the network node N5 returning a response comprising verification information; and after receiving the response from the network node N5 itself or a response from the other network node, the request end C5 of the network node N5 synchronizing the data D5. The data synchronization method designed in the present invention is simple to implement, and by using a request-response mode, the synchronization of a single node and a plurality of nodes is covered in a recursive manner, thereby reducing the system complexity and improving the system reliability.

Description

A decentralized node data synchronization method and network system

[0001] This application claims priority to the Chinese patent application with application number 202311257361.X filed on September 26, 2023, entitled “A decentralized node data synchronization method and network system”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to a data synchronization method and a network system for decentralized nodes, belonging to the field of communication technology or network technology. Background Art

[0003] In a distributed system, data stored on different nodes must be retrieved, modified, added, and deleted. Existing data synchronization technologies are applicable to distributed systems that generally involve master and slave nodes. During the data synchronization process, eventual data consistency can generally be guaranteed. Specifically:

[0004] 1. Existing technical solutions generally use a master-slave node approach, with the master node serving as the central database. When data changes occur on a slave node, the changes are sent to the master node, which then reflects the changes to other nodes. This master-slave node approach has complex network interactions, and the real-time performance of data synchronization is affected by the synchronization of the master node, resulting in poor real-time performance.

[0005] 2. When the master node fails, the network needs to re-elect the master node, and the election mechanism is complex and difficult to implement. The election time is long when a failure occurs, and the complex mechanism leads to low reliability and consumes a lot of node resources.

[0006] 3. Existing data synchronization methods can generally ensure eventual data consistency or loose consistency, that is, they do not require strict data consistency and allow data in different nodes to be inconsistent within a certain period of time.

[0007] Summary of the Invention

[0008] The technical problem to be solved by the present invention is to overcome the shortcomings of the existing technology and solve the data synchronization problem of decentralized nodes.

[0009] The purpose of the present invention is achieved through the following technical solutions:

[0010] A data synchronization method for decentralized nodes, wherein the nodes include data D, a requesting end C, and a responding end S. When only one node is online, the data synchronization method includes:

[0011] The requester C of the node sends a request, which includes its own data D;

[0012] After receiving the request, the responder S of the node verifies the data D and returns a response including the verification information;

[0013] After receiving the response, the requesting end C of the node synchronizes the data D.

[0014] A data synchronization method for decentralized nodes, each node includes data D, a request end C, and a response end S;

[0015] When multiple nodes are online, for any two nodes, namely node N1 and node N2, the data synchronization method includes:

[0016] The requester C1 of node N1 sends a request to node N2, which includes its own data D1;

[0017] After receiving the request, the responder S2 of node N2 verifies the data D1 and returns a response including verification information;

[0018] After receiving the response, the requester C1 of the node N1 synchronizes the data D1.

[0019] A data synchronization method for decentralized nodes, each node includes data D, a request end C, and a response end S;

[0020] When multiple nodes are online, data D3 of at least one node N3 is not empty, and data D4 of at least one node N4 is empty; the node N4 with empty data D4 does not send a request;

[0021] The data synchronization method between node N3 and node N4 includes:

[0022] The requester C1 of node N3 sends a request to node N4, which includes its own data D3;

[0023] After receiving the request, the responder S4 of the node N4 verifies the data D3 and returns a response including the verification information;

[0024] After receiving the response, the requester C3 of the node N3 synchronizes the data D3.

[0025] In one embodiment of the present invention, data D of only one node among the multiple nodes is not empty.

[0026] In one embodiment of the present invention, the request parameter type is: container <data item>.

[0027] In one embodiment of the present invention, a node that receives the request verifies the data item by item.

[0028] In one embodiment of the present invention, the response parameter type is: container <data item>.

[0029] In one embodiment of the present invention, the returned response includes verification exception information.

[0030] In one embodiment of the present invention, the communication layer dynamically senses whether a node is online or offline.

[0031] In one embodiment of the present invention, the communication layer uses UDP to achieve dynamic perception. After the node comes online and starts running, it sends its own heartbeat data to a preset multicast address; if the node's heartbeat is not received for more than a specified period of time, the node is determined to be offline.

[0032] In one embodiment of the present invention, when the communication layer uses a topic-based message queue to implement dynamic perception, each node publishes heartbeat data to the same heartbeat topic and subscribes to the topic.

[0033] In one embodiment of the present invention, the communication layer uses the DDS data distribution service, and each node processes the node online and offline events sent by the DDS.

[0034] In one embodiment of the present invention, after receiving the request, if the node determines that the synchronization data is abnormal, the abnormal data is marked in the response.

[0035] A decentralized node network system includes at least one online network node N5, each network node includes data D, a request end C, and a response end S;

[0036] Network node N5 uses the following method to synchronize data:

[0037] The requester C5 of the network node N5 sends a request, which includes its own data D5;

[0038] After receiving a request from itself or another network node, the responder S5 of the network node N5 verifies the data in the request and returns a response including verification information;

[0039] After receiving a response from itself or another network node, the requester C5 of the network node N5 synchronizes data D5.

[0040] A decentralized node network system includes at least two online network nodes, each of which includes data D, a request end C, and a response end S;

[0041] If the data D6 of at least one online network node N6 is not empty, and the data D7 of at least one online network node N7 is empty, then the online network node N7 whose data D7 is empty does not send a request;

[0042] Network node N6 uses the following method to synchronize data:

[0043] The requester C6 of the network node N6 sends a request, which includes its own data D6;

[0044] After receiving a request from itself or another network node, the responder S6 of the network node N6 verifies the data in the request and returns a response including verification information;

[0045] After receiving a response from itself or another network node, the requester C6 of the network node N6 synchronizes data D6.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] (1) The data synchronization method designed by the present invention is simple to implement and uses a request-response method in a recursive form to cover the synchronization of single nodes and multiple nodes, thereby reducing system complexity and improving system reliability.

[0048] (2) The data synchronization method designed by the present invention can reduce the network load by setting empty data nodes, and has good adaptability in networks with different node sizes.

[0049] (3) The data synchronization method designed by the present invention includes a verification feedback link, which can meet the data fault tolerance requirements.

[0050] (4) The data synchronization method designed by the present invention has good support for heterogeneous data and good applicability.

[0051] (5) The present invention can ensure timely synchronization of data within the system, and the data change information of any node is immediately updated in all nodes, ensuring strong data consistency.

[0052] (6) The data synchronization method designed by the present invention has low requirements on the communication layer and can be replaced by a communication middleware based on topic publishing and subscription communication. It has low development difficulty and good testability.

[0053] (7) The present invention can complete synchronization in a decentralized manner without relying on a single database, without the need for prior deployment or configuration, and can be used immediately at any node. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] FIG1 is a flow chart of the steps of the method of the present invention;

[0055] FIG2 is a schematic diagram of a request from node N1 to node N2;

[0056] FIG3 is a schematic diagram of the response of node N2 to node N1;

[0057] FIG4 is a schematic diagram showing the state of the data D1 after the node N1 processes the response;

[0058] FIG5 is a schematic diagram showing the state of the data D2 after the node N2 processes the response;

[0059] FIG6 is a schematic diagram of the data D1 state after the node N1 processes its own request and feeds back a response;

[0060] FIG7 is a schematic diagram of a request and response between node N2 and node N1 when data D1 of node N1 changes;

[0061] Figure 8 shows the dynamic perception results of other nodes after node N1 goes offline. DETAILED DESCRIPTION

[0062] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0063] Example 1:

[0064] A data synchronization method for decentralized nodes, wherein node N includes data D, a requesting end C, and a responding end S. The data synchronization method includes:

[0065] Taking nodes N1 and N2 (hereinafter referred to as N1 and N2) in a network as an example, the data synchronization process when nodes dynamically sense a node's online presence is described. As shown in Figure 1, during the synchronization process for node N1's data D1, the requester C1 of node N1 sends a request to node N2, including its own data D1. Upon receiving the request, the responder S2 of node N2 verifies data D1 and returns a response including the verification information. Upon receiving the response, the requester C1 of node N1 synchronizes data D1.

[0066] The synchronization process includes:

[0067] 1. N1 discovers that N2 has come online and sends a request to N2. This request includes its own data D1 (request parameter type: container <data item>). As shown in Figure 2, node N1 sends a request to node N2, and the requesting client C1 sends a request to the responding client S2. Data D1 represents a collection of data items, or a dataset.

[0068] 2. After receiving N1's request, N2 verifies each item in D1 and returns a response containing verification error information (response parameter type: container <data item>). As shown in Figure 3, after node N2 receives N1's request, responder S2 sends a response to requester C1. N2 records D1's information and identifies the verification results of the data in D1.

[0069] 3. After receiving the response, N1 synchronizes data D1. As shown in Figure 4, node N1 receives the response from S2 and identifies the verification result of its own D1 data in N2.

[0070] Similarly, N2 discovers that N1 is online, sends a request, and receives a response. After synchronization, the data on N1 and N2 are consistent. As shown in Figure 5, after the consistent synchronization process, N2 data D2 is synchronized on N1 and N2 respectively. After synchronization, the data on nodes N1 and N2 are consistent.

[0071] When there are more nodes in the network, they are all synchronized according to the above synchronization process, and the data of each node is consistent.

[0072] Synchronization includes data addition, deletion, and modification.

[0073] For example, if only node N1 exists in the network, changes to N1's own data set are also synchronized using this request-response method, consistent with the synchronization between different nodes. During a self-update, N1's requester, C1, sends a request to its own responder, S1, and receives a response, completing the update of its own data. As shown in Figure 6, this consistent synchronization process is also used for updating a node's own data. Node N1's requester, C1, sends a request to its own responder, S1, and receives a response from S1, completing the update of node data D1.

[0074] When a dataset change notifies other nodes to update, the request and response processing is consistent with the previous process. As shown in Figure 7, this synchronization process also covers D1 data change notifications to other nodes to update, and the request and response processing is consistent with the previous process.

[0075] When a node dynamically detects that another node has gone offline, it does not need to go through a request-response process, but directly deletes the offline node's data. As shown in Figure 8, when a node dynamically detects that another node has gone offline, it does not need to go through a request-response process, but directly deletes the offline node's data.

[0076] During data synchronization, the communication layer dynamically detects the online and offline status of nodes, including the following methods:

[0077] 1. When the communication layer uses UDP to achieve dynamic perception, each node sends its own heartbeat data to the preset multicast address after startup. Each node joins the multicast address to receive the heartbeat data.

[0078] Heartbeat data format: node unique identifier (8 bytes) + extension (n bytes)

[0079] Node unique identifier: composed of the device IP (4 bytes) where the node is running + the node process PID (4 bytes)

[0080] Extensions: These are serialized key-value pairs. These can be empty by default, but can be expanded to include information such as CPU, memory, and runtime for easier monitoring. For example, node inactivity duration: If a node's heartbeat is not received for a specified period, the node is considered offline.

[0081] The extension item is a set of map key-value pairs, where the key is defined as string and the value is defined as variant to support various types;

[0082] By traversing the keys of the map, when the extended content of the extension item is increased, it does not affect the call.

[0083] When the extension item is sent, it is serialized to obtain a byte sequence. The specific serialization steps are: traversing the map key, writing the key and the corresponding value into the byte sequence in turn, and deciding whether to compress the byte sequence before sending based on the characteristics of the byte sequence (compression is optional and is not used in scenarios with small data volumes or that are sensitive to computation).

[0084] The extension receives a byte sequence and deserializes it to obtain a map. The specific deserialization steps are: if the byte sequence is compressed, it is first decompressed, and the key and corresponding value are taken out from the byte sequence in sequence and written into the map.

[0085] 2. When the communication layer uses a topic-based message queue to implement dynamic perception, each node publishes heartbeat data to the same heartbeat topic and subscribes to the topic. The heartbeat data format is the same as above.

[0086] 3. When the communication layer uses the DDS data distribution service, each node processes the node online and offline events sent by DDS without sending additional heartbeat data.

[0087] Methods for handling exceptions during data synchronization include:

[0088] After receiving the request, if the node determines that the synchronization data is abnormal, it should mark the abnormal data in the response, and the marking content is a data item.

[0089] For each data item, the node data record content can be expanded, mainly including:

[0090] Data item identifier, unique identifier of the node where the data item is located, and extension item.

[0091] If data update time is included, the time is recorded in the local time of the node. All nodes are peer-to-peer, and there is no unified time synchronization. See Table 1 for an example of N1 data.

[0092] Table 1

[0093] After N2 sends a synchronization request to N1, N1 determines that the data "test1 / data1" is abnormal and includes "test1 / data1" in its response. Subsequent nodes then process the data "test1 / data1" according to the algorithm based on the data update time.

[0094] Example 2:

[0095] A decentralized node data synchronization method differs from Example 1 in that data synchronization uses dataless nodes. Nodes without their own data do not need to send requests to other nodes. Using dataless nodes further reduces network load and system complexity. For example, one node in the system records all data, while all other nodes are dataless nodes.

[0096] Example 3:

[0097] In the expert interpretation system, multiple interpretation devices are operated simultaneously by multiple experts. Without the need for a database, each interpretation device is treated as a node, and the data synchronization method described in Example 1 is used to achieve real-time synchronization of criteria. The data D of each node is described using XML, and the expert interpretation system uses this synchronization method to automatically update the criteria.

[0098] Example 4:

[0099] In a distributed file storage system, each file storage device acts as a node, and the data synchronization method described in Example 1 is used to synchronize file directory information. Each node's data item, D, is identified by a file path / file name. By querying the file directory information, a device can retrieve files from a specified node.

[0100] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

[0101] Although the present invention has been disclosed above in terms of preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solutions of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the scope of protection of the technical solutions of the present invention.

Claims

1. A decentralized node data synchronization method, characterized in that: The nodes include data D, request end C, and response end S. When only one node is online, the data synchronization method includes: The request end C of the node sends a request, which includes its own data D; After receiving the request, the responder S of the node verifies the data D and returns a response including verification information; After receiving the response, the requesting end C of the node synchronizes the data D.

2. A decentralized node data synchronization method, characterized in that: Each node includes data D, request end C, and response end S; When multiple nodes are online, for any two nodes, namely, node N1 and node N2, the data synchronization method includes: The request end C1 of node N1 sends a request to node N2, and the request includes its own data D1; After receiving the request, the responder S2 of the node N2 verifies the data D1 and returns a response including verification information; After receiving the response, the requesting end C1 of the node N1 synchronizes the data D1.

3. A decentralized node data synchronization method, characterized in that: Each node includes data D, request end C, and response end S; When multiple nodes are online, data D3 of at least one node N3 is not empty, and data D4 of at least one node N4 is empty; Node N4 whose data D4 is empty does not issue a request; The data synchronization method of the node N3 and the node N4 includes: The request end C1 of the node N3 sends a request to the node N4, and the request includes its own data D3; After receiving the request, the responder S4 of the node N4 verifies the data D3 and returns a response including verification information; After receiving the response, the requesting end C3 of the node N3 synchronizes the data D3.

4. The data synchronization method according to claim 3, characterized in that: Data D of only one node among the multiple nodes is not empty.

5. The data synchronization method according to any one of claims 1 to 4, characterized in that: The request parameter type is: container <data item>.

6. The data synchronization method according to any one of claims 1 to 4, characterized in that: The node that receives the request verifies the data item by item.

7. The data synchronization method according to any one of claims 1 to 4, characterized in that: The response parameter type is: container <data item>.

8. The data synchronization method according to any one of claims 1 to 4, characterized in that: The returned response includes validation exception information.

9. The data synchronization method according to any one of claims 1 to 4, characterized in that: Dynamically sense when nodes are online or offline at the communication layer.

10. The data synchronization method according to claim 9, characterized in that: The communication layer uses UDP to achieve dynamic perception. After the node is online and starts running, it sends its own heartbeat data to the preset multicast address. If the heartbeat of the node is not received for more than a specified period of time, the node is considered offline.

11. The data synchronization method according to claim 9, characterized in that: When the communication layer uses a topic-based message queue to achieve dynamic perception, each node publishes heartbeat data to the same heartbeat topic and subscribes to the topic.

12. The data synchronization method according to claim 9, characterized in that: The communication layer uses the DDS data distribution service, and each node processes the node online and offline events issued by DDS.

13. The data synchronization method according to any one of claims 1 to 4, characterized in that: After receiving the request, if the node determines that the synchronization data is abnormal, it will mark the abnormal data in the response.

14. A decentralized node network system, characterized in that: It includes at least one online network node N5, each of which includes data D, a request end C, and a response end S; Network node N5 uses the following method to synchronize data: The request end C5 of the network node N5 sends a request, which includes its own data D5; After receiving a request from itself or another network node, the responder S5 of the network node N5 verifies the data in the request and returns a response including verification information; After receiving a response from itself or other network nodes, the requester C5 of the network node N5 synchronizes data D5.

15. A decentralized node network system, characterized in that: It includes at least two online network nodes, each of which includes data D, a request end C, and a response end S; If data D6 of at least one online network node N6 is not empty, and data D7 of at least one online network node N7 is empty, then the online network node N7 whose data D7 is empty does not send a request; Network node N6 uses the following method to synchronize data: The request end C6 of the network node N6 sends a request, which includes its own data D6; After receiving a request from itself or another network node, the responder S6 of the network node N6 verifies the data in the request and returns a response including verification information; After receiving a response from itself or other network nodes, the requester C6 of the network node N6 synchronizes data D6.

Images