Multi-node data synchronization optimization method and device, equipment and medium

Abstract

The invention relates to the technical field of data synchronization, and discloses a multi-node data synchronization optimization method and device, equipment and a medium, and the method comprises the steps: responding to at least one data node needing to carry out data synchronization, and determining a scene corresponding to the data synchronization; in response to the situation that the scene is power-on starting, data synchronization between the source node and the target node is completed based on dynamic difference comparison; in response to the situation that the scene is node restart, data synchronization of the target node is completed based on parallelism and fragmentation; and in response to the situation that the scene is local node synchronization, completing data synchronization of the target node based on the dynamic optimal path. According to the scheme of the invention, the scene of data synchronization is determined, and different data synchronization modes are adopted for different scenes, so that a multi-mode adaptive synchronization mechanism suitable for multiple nodes is realized, and the synchronization efficiency and the resource utilization rate are improved on the premise of ensuring the data consistency.

Description

Technical Field

[0001] This invention relates to the field of data synchronization technology, and specifically to a multi-node data synchronization optimization method, apparatus, device, and medium. Background Technology

[0002] In domestically developed management platforms, such as the OpenBMC management platform, data synchronization across multiple nodes (≥3 nodes) relies on a centralized architecture. Synchronization latency increases exponentially with the number of nodes, and it only supports unidirectional synchronization from the central node to slave nodes, unable to directly communicate across nodes. This architecture suffers from high synchronization latency, limited communication, and efficiency bottlenecks.

[0003] In related technologies, existing multi-node data synchronization methods employ a hybrid mode of broadcast, unicast, and multicast. While this approach alleviates the latency issues of centralized architectures, its efficiency remains insufficient when dealing with large amounts of VPD (Vital Product Data) (such as firmware and configuration logs). It is inefficient when transmitting large volumes of data, and the broadcast mode may cause network congestion as the number of nodes increases. Furthermore, fixed synchronization strategies cannot be adjusted according to real-time load, resulting in inefficiencies in these application scenarios and ultimately reducing the overall data synchronization efficiency. Summary of the Invention

[0004] In view of this, the present invention provides a method, apparatus, device, and medium for optimizing data synchronization in a multi-node environment, in order to solve the technical problem of low data synchronization efficiency in multi-node application environments.

[0005] In a first aspect, the present invention provides a multi-node data synchronization optimization method, the method comprising: in response to the need for data synchronization of at least one data node, determining a scenario corresponding to the data synchronization; in response to the scenario being power-on startup, completing data synchronization between the source node and the target node based on dynamic difference comparison; in response to the scenario being local node synchronization, completing data synchronization of the target node based on parallelism and sharding; and in response to the scenario being node restart, completing data synchronization of the target node based on dynamic optimal path.

[0006] In conjunction with the first aspect, in one possible implementation of the first aspect, data synchronization between the source node and the target node is completed based on dynamic difference comparison, including: comparing the version numbers between the source node and the target node; in response to the version number inconsistency, performing difference comparison to determine the difference package of the changed fields; and completing the data synchronization between the source node and the target node based on the difference package.

[0007] In conjunction with the first aspect, in one possible implementation of the first aspect, a difference comparison is performed to determine the difference package of the changed fields, including: The structured data is compared field by field to generate a first difference packet containing the changed field names and new values; a second difference packet corresponding to the binary block is determined based on the difference algorithm; the first and second difference packets are used together as the difference packet for the changed fields.

[0008] In conjunction with the first aspect, in one possible implementation of the first aspect, data synchronization of the target node is achieved based on parallelism and fragmentation, including: splitting the data to be synchronized into multiple independent fragments according to a preset priority; and receiving each fragment in parallel along the link corresponding to the preset priority to complete the data synchronization of the target node.

[0009] In conjunction with the first aspect, in one possible implementation of the first aspect, the data to be synchronized is split according to a preset priority to form multiple independent fragments, including: splitting the data to be synchronized based on the data type to determine the data classification result; and forming multiple independent fragments based on the preset correspondence between the data classification result and the priority.

[0010] In conjunction with the first aspect, one possible implementation of the first aspect involves completing the data synchronization of the target node based on a dynamic optimal path, including: determining the optimal path based on network load and node status; determining the data propagation method of the target node based on network load; and completing the data synchronization of the target node based on the optimal path and data propagation method.

[0011] In conjunction with the first aspect, in one possible implementation of the first aspect, the data propagation method of the target node is determined based on network load, including: multicast in response to low network load; unicast in response to high network load; and unicast, multicast, and broadcast methods sequentially in response to network load failure.

[0012] Secondly, the present invention provides a multi-node data synchronization optimization device, comprising: a scenario determination module, used to determine a scenario corresponding to data synchronization in response to at least one data node needing to perform data synchronization; a synchronization startup module, used to complete data synchronization between a source node and a target node based on dynamic difference comparison in response to a power-on startup scenario; a local synchronization module, used to complete data synchronization of the target node based on parallelism and sharding in response to a node restart scenario; and a synchronization restart module, used to complete data synchronization of the target node based on a dynamic optimal path in response to a local node synchronization scenario.

[0013] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the multi-node data synchronization optimization method described in the first aspect or any corresponding embodiment thereof.

[0014] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the multi-node data synchronization optimization method described in the first aspect or any corresponding embodiment thereof.

[0015] The technical solution of this invention has the following advantages: This invention provides a multi-node data synchronization optimization method, apparatus, device, and medium. The method includes: in response to at least one data node requiring data synchronization, determining a scenario corresponding to the data synchronization; in response to the scenario of power-on startup, completing data synchronization between the source node and the target node based on dynamic difference comparison; in response to the scenario of node restart, completing data synchronization of the target node based on parallelism and sharding; and in response to the scenario of local node synchronization, completing data synchronization of the target node based on a dynamic optimal path. In this process, by determining the data synchronization scenario and adopting different data synchronization methods for different scenarios, a flexible data synchronization strategy is achieved in multi-node application environments. This realizes a multi-mode adaptive synchronization mechanism suitable for multiple nodes, improving synchronization efficiency and resource utilization while ensuring data consistency. Attached Figure Description

[0016] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a multi-node data synchronization optimization method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of an application scenario provided by an embodiment of the present invention; Figure 3 This is a structural block diagram of a multi-node data synchronization optimization device provided according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] According to an embodiment of the present invention, a method for optimizing data synchronization of multiple nodes is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0020] This embodiment provides a multi-node data synchronization optimization method, such as... Figure 1 As shown, the method includes the following steps: S101. In response to the need for data synchronization at at least one data node, determine the scenario corresponding to the data synchronization.

[0021] Specifically, the data synchronization system architecture between multiple nodes is as follows: Figure 2 As shown, the interaction relationships between four nodes are illustrated.

[0022] Specifically, the scenarios corresponding to data synchronization include: power-on startup, node restart, and partial node synchronization. Among them, the power-on startup scenario involves all or some nodes starting up simultaneously, the node restart scenario includes single node restart / upgrade, and the partial node synchronization scenario refers to data synchronization between some nodes.

[0023] S102. In response to the scenario of power-on startup, data synchronization between the source node and the target node is completed based on dynamic difference comparison.

[0024] Specifically, in power-on startup scenarios, a large number of nodes start simultaneously, and broadcasting full data requests can cause a broadcast storm. Instead of broadcasting the entire dataset, nodes broadcast requests carrying their local data version numbers. Target nodes, upon receiving these requests, compare version differences and only return the changed fields. This achieves lightweight broadcast triggering and incremental data return, reducing bandwidth usage and synchronization time.

[0025] S103. Responding to the scenario of local node synchronization, based on parallelism and sharding, the data synchronization of the target node is completed.

[0026] Specifically, during local node synchronization, a fragment request is multicast to the target node group. The receiving end verifies the fragment hash value in real time. If a fragment fails to be verified, only that fragment is retransmitted, without having to roll back the entire dataset.

[0027] S104. In response to the scenario of node restart, data synchronization of the target node is completed based on the dynamic optimal path.

[0028] Specifically, in the node restart scenario, the initiating node actively detects the network latency of neighboring nodes, dynamically selects the optimal path, splits the data into independent shards, and pulls them in parallel using multiple threads.

[0029] This invention provides a multi-node data synchronization optimization method, apparatus, device, and medium. The method includes: in response to at least one data node requiring data synchronization, determining a scenario corresponding to the data synchronization; in response to the scenario of power-on startup, completing data synchronization between the source node and the target node based on dynamic difference comparison; in response to the scenario of node restart, completing data synchronization of the target node based on parallelism and sharding; and in response to the scenario of local node synchronization, completing data synchronization of the target node based on a dynamic optimal path. In this process, by determining the data synchronization scenario and adopting different data synchronization methods for different scenarios, a flexible data synchronization strategy is achieved in multi-node application environments. This realizes a multi-mode adaptive synchronization mechanism suitable for multiple nodes, improving synchronization efficiency and resource utilization while ensuring data consistency.

[0030] In one optional implementation, data synchronization between the source node and the target node is achieved based on dynamic difference comparison, including: Compare the version numbers between the source node and the target node; in response to the version number inconsistency, perform a difference comparison to determine the difference package for the changed fields; based on the difference package, complete the data synchronization between the source node and the target node.

[0031] Specifically, in the power-on startup scenario, comparing the version numbers of the source and target nodes involves broadcasting the version numbers of the source and target nodes for comparison. This version number comparison serves as a pre-check mechanism, avoiding invalid full data transmissions.

[0032] Specifically, data synchronization between the source node and the target node based on the difference packet refers to completing the data synchronization between the source node and the target node using an incremental synchronization protocol based on the determined difference data, i.e., the difference packet.

[0033] In one alternative implementation, a difference comparison is performed to determine the difference package of the changed fields, including: The structured data is compared field by field to generate a first difference packet containing the changed field names and new values; a second difference packet corresponding to the binary block is determined based on the difference algorithm; the first and second difference packets are used together as the difference packet for the changed fields.

[0034] Specifically, structured data includes JSON, XML, and other structured data. For JSON, XML, and other structured data, a field-by-field comparison is performed, and the generated first difference packet contains the changed field names and new values.

[0035] Specifically, differential algorithms include binary differential algorithms such as Bsdiff, which are used to generate a second differential packet for large binary blocks.

[0036] In one alternative implementation, data synchronization of the target node is achieved based on parallelism and sharding, including: The data to be synchronized is split into multiple independent fragments according to a preset priority; each fragment is received in parallel along the link corresponding to the preset priority to complete the data synchronization of the target node.

[0037] Specifically, data synchronization with the target node involves receiving each data fragment in parallel along links corresponding to preset priorities. This means allocating high-priority fragments to low-latency links, such as direct connections between nodes, and using multi-threaded concurrent reception of multiple fragments. When receiving multiple fragments concurrently using multi-threaded reception, real-time verification ensures data integrity.

[0038] In one optional implementation, the data to be synchronized is split according to a preset priority to form multiple independent fragments, including: Based on the data type, the data to be synchronized is split and the data classification results are determined; based on the preset correspondence between the data classification results and priorities, multiple independent shards are formed.

[0039] Specifically, data is divided into different logical priorities according to type, and data of the same priority are grouped together into an independent transmission unit, i.e., a fragment. The preset correspondence between the classification results and priorities is shown in Table 1, which exemplarily outputs the correspondence between priority, data type, and division criteria.

[0040] Table 1

[0041] In one optional implementation, data synchronization of the target node is completed based on a dynamically optimal path, including: Based on network load and node status, determine the optimal path; based on network load, determine the data propagation method for the target node; based on the optimal path and data propagation method, complete the data synchronization of the target node.

[0042] Specifically, determining the optimal path based on network load and node status involves inputting metrics such as link quality, network load, and node status into a trained weighted scoring model. This model scores all available candidate paths and selects the path with the highest overall score as the optimal path.

[0043] Specifically, achieving data synchronization of the target node based on the optimal path and data propagation method means using parallel multi-threading to pull data based on a determined optimal path and data propagation method, thereby achieving data synchronization of the target node.

[0044] In one optional implementation, the data propagation method of the target node is determined based on network load, including: When the network load is low, multicast is used; when the network load is high, unicast is used; when the network load is faulty, unicast, multicast, and broadcast are used sequentially.

[0045] Specifically, when the network load is low, multicast is prioritized for communication between nodes. When the network load is high, communication between nodes is switched to unicast. In the event of a network failure, unicast is attempted first; if unicast fails, multicast is switched to; if multicast succeeds, the process is marked as complete; otherwise, broadcast is reverted to broadcast.

[0046] This embodiment also provides a multi-node data synchronization optimization device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0047] This embodiment provides a multi-node data synchronization optimization device, such as... Figure 3 As shown, it includes: The scenario determination module 201 is used to determine the scenario corresponding to the data synchronization in response to the need for data synchronization at least one data node. For details, please refer to the description of step S101 in the above embodiments, which will not be repeated here.

[0048] The synchronization module 202 is activated in response to a power-on scenario to complete data synchronization between the source node and the target node based on dynamic difference comparison. For details, please refer to the description of step S102 in the above embodiments, which will not be repeated here.

[0049] The local synchronization module 203 is used to respond to a node restart in a scenario, and to complete the data synchronization of the target node based on parallelism and sharding. For details, please refer to the relevant description of step S103 in the above embodiments, which will not be repeated here.

[0050] The synchronization module 204 is restarted to respond to the scenario of local node synchronization and complete the data synchronization of the target node based on the dynamic optimal path. For details, please refer to the relevant description of step S104 in the above embodiments, which will not be repeated here.

[0051] In this embodiment, the multi-node data synchronization optimization device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0052] This invention also provides a computer device having the above-described features. Figure 3 The multi-node data synchronization optimization device shown is illustrated.

[0053] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 4 As shown, the computer device includes one or more processors 301, memory 302, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 4 Take processor 301 as an example.

[0054] Processor 301 may be a central processing unit, a network processor, or a combination thereof. Processor 301 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0055] The memory 302 stores instructions executable by at least one processor 301 to cause the at least one processor 301 to perform the method shown in the above embodiments.

[0056] Memory 302 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, memory 302 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, memory 302 may optionally include memory remotely located relative to processor 301, and this remote memory may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0057] The memory 302 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 302 may also include combinations of the above types of memory. The computer device also includes a communication interface 303 for communicating with other devices or communication networks.

[0058] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0059] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A multi-node data synchronization optimization method, characterized in that, The method comprises: in response to at least one data node needing data synchronization, determining a scenario corresponding to the data synchronization; in response to the scenario being power-on start, completing data synchronization between a source node and a target node based on dynamic difference comparison; in response to the scenario being local node synchronization, completing data synchronization of the target node based on parallelism and sharding; in response to the scenario being node restart, completing data synchronization of the target node based on a dynamic optimal path.

2. The method of claim 1, wherein, The method comprises: comparing version numbers between the source node and the target node; in response to the version numbers being inconsistent, performing difference comparison to determine a difference package of changed fields; based on the difference package, completing data synchronization between the source node and the target node.

3. The method of claim 2, wherein, The method comprises: performing field-by-field comparison on structured data to generate a first difference package containing changed field names and new values; based on a difference algorithm, determining a second difference package corresponding to binary blocks; using the first difference package and the second difference package as the difference package of changed fields.

4. The method of claim 1, wherein, The method comprises: splitting data to be synchronized according to a preset priority to form multiple independent shards; parallelly receiving each shard along a link corresponding to the preset priority to complete data synchronization of the target node.

5. The method of claim 4, wherein, The method comprises: based on the type of data, splitting data to be synchronized to determine a data classification result; based on a preset correspondence between the data classification result and the priority, forming multiple independent shards.

6. The method of claim 1, wherein, The method comprises: based on network load and node status, determining an optimal path; based on the network load, determining a data propagation mode of the target node; based on the optimal path and the data propagation mode, completing data synchronization of the target node.

7. The method of claim 6, wherein, The method comprises: in response to the network load being low, using a multicast mode; in response to the network load being high, using a unicast mode; in response to the network load being faulty, sequentially using unicast, multicast, and broadcast modes.

8. A multi-node data synchronization optimization apparatus, characterized by, The device comprises: a scenario determination module configured to determine a scenario corresponding to data synchronization in response to at least one data node needing data synchronization; a start synchronization module configured to complete data synchronization between a source node and a target node based on dynamic difference comparison in response to the scenario being power-on start; a local synchronization module configured to complete data synchronization of the target node based on parallelism and sharding in response to the scenario being local node synchronization; a restart synchronization module configured to complete data synchronization of the target node based on a dynamic optimal path in response to the scenario being node restart.

9. A computer device, comprising: The device comprises: A memory and a processor, which are connected in communication with each other, the memory has stored computer instructions, and the processor executes the computer instructions to perform the multi-node data synchronization optimization method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium has stored computer instructions for causing a computer to perform the multi-node data synchronization optimization method of any one of claims 1 to 7.

Images