A data synchronization method, device and equipment of a distributed system and a storage medium

Abstract

The embodiment of the present specification relates to the computer technical field, provide a kind of data synchronization method, device and equipment of distributed system and storage medium, the method comprises: first, according to update data request, data update is carried out to source node database, and the execution log of source node database in updating process is obtained, then using analysis server obtains the data synchronization optimal scheme that source node to target node communication time consumption is shortest, finally, according to the execution log of source node and data synchronization optimal scheme, data synchronization is carried out to target node database. Through the embodiment of the present specification, data synchronization efficiency can be improved.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a data synchronization method, apparatus, device, and storage medium for a distributed system. Background Technology

[0002] Data synchronization refers to the real-time or periodic updating and matching of data from two or more different locations or systems to maintain data consistency. With the development of business and technology, data synchronization technology is evolving towards clustering. Inter-cluster data synchronization technology is a fundamental technology for disaster recovery solutions, and research into efficient and consistent data synchronization methods between clusters and data centers is of great and far-reaching significance.

[0003] Existing distributed cluster data synchronization schemes involve: first, updating the source cluster's database; second, configuring Kafka producers and Kafka consumers for other clusters requiring parameter updates; third, querying the corresponding database through a parameter synchronization initiator, converting the query results into target strings, compressing the results, and sending them to message consumers via a distributed messaging system; and fourth, the parameter synchronization receiver retrieving the necessary updated parameter data through the distributed messaging system, decompressing it, converting the data into objects, performing add or modify operations, and then updating the target cluster's data tables. This scheme creates numerous objects and performs extensive encoding, parsing, and assignment operations during synchronization. When there are multiple target cluster databases, data synchronization efficiency is low. Furthermore, its master-slave architecture leads to extreme dependence on the master node; if the source cluster's system or corresponding database fails, the parameter synchronization function becomes unusable, impacting the distributed cluster's data synchronization efficiency. Therefore, a technical solution is urgently needed to address the low data synchronization efficiency problem in existing distributed cluster technologies. Summary of the Invention

[0004] In view of the above-mentioned problems in the prior art, the purpose of the embodiments of this specification is to provide a data synchronization method, apparatus, device and storage medium for a distributed system, so as to solve the problem of low data synchronization efficiency in the distributed cluster in the prior art.

[0005] To solve the above-mentioned technical problems, the specific technical solutions of the embodiments in this specification are as follows:

[0006] On one hand, embodiments of this specification provide a data synchronization method for a distributed system, wherein the distributed system includes several clusters and an analysis server, each cluster is configured with at least one source node and several target nodes, and the method is applied to the source nodes, including:

[0007] Receive an update data request, update the source node database according to the update data request, and obtain the execution log of the source node database;

[0008] Send a data synchronization sequence acquisition request to the analysis server;

[0009] The system receives the optimal data synchronization scheme sent by the analysis server, wherein the optimal data synchronization scheme includes the data synchronization path with the shortest communication time from the source node to the target node.

[0010] Data synchronization is performed on the target node database based on the execution logs of the source node database and the optimal data synchronization scheme.

[0011] Furthermore, the step of synchronizing data to the target node database based on the execution logs of the source node database and the optimal data synchronization scheme includes:

[0012] The execution logs of the source node database are added to the relay logs of the target node database in the order of data synchronization of the target nodes. Each target node database is updated according to the events in the relay logs.

[0013] Furthermore, the method also includes:

[0014] Obtain the updated data from the source node database and the target node database;

[0015] The updated data in the target node database is compared with the updated data in the source node database to obtain the comparison result;

[0016] Based on the comparison results, determine whether the data synchronization of the target node is abnormal;

[0017] If so, an alarm notification is sent to the target node where the data synchronization error occurred, and the data synchronization of the target node where the data synchronization error occurred is re-performed.

[0018] Furthermore, the method also includes:

[0019] When the database of each target node is updated according to the events in the relay log, the execution process of the target node is monitored to obtain the execution status of the target node;

[0020] If the execution status of the target node is failed and the number of failures is less than the preset number, then the target node will be resynchronized.

[0021] On the other hand, embodiments of this specification provide a data synchronization method for a distributed system, wherein the distributed system includes several clusters and an analysis server, each cluster is configured with at least one source node and several target nodes, and the method is applied to the analysis server, including:

[0022] Receive the data synchronization order acquisition request sent by the source node;

[0023] According to the data synchronization order, the network status of the source node and the target node is obtained by requesting;

[0024] The most efficient communication transmission method is matched based on the network status.

[0025] Predict the communication time between nodes under the aforementioned communication transmission method, and generate the data synchronization path with the shortest overall communication time based on the communication time.

[0026] Based on the data synchronization path, the optimal data synchronization solution is obtained;

[0027] Send the optimal data synchronization solution to the source node.

[0028] Furthermore, generating the data synchronization path with the shortest overall communication time based on the communication time includes:

[0029] S1: Starting from the source node;

[0030] S2: Find the target node with the shortest communication time with the source node as the second node, and use the second node as the new starting point to find the target node with the shortest communication time with the second node as the third node;

[0031] S3: Calculate the communication time of the source node through the second node to the third node and calculate the communication time of the source node to the third node;

[0032] S4: Select the path with the shortest communication time as the shortest path from the starting point to the third node;

[0033] S5: Using the third node as the new starting point, repeat steps S2-S4 until the shortest path from the source node to all target nodes is determined.

[0034] On the other hand, embodiments of this specification provide a data synchronization device for a distributed system, wherein the distributed system includes several clusters and an analysis server, each cluster is configured with at least one source node and several target nodes, and the device is applied to the source nodes, including:

[0035] The data update module is used to receive update data requests, update the source node database according to the update data requests, and obtain the execution log of the source node database;

[0036] The synchronization order acquisition module is used to send a data synchronization order acquisition request to the analysis server;

[0037] The first receiving module is used to receive the optimal data synchronization scheme sent by the analysis server, wherein the optimal data synchronization scheme includes the data synchronization path with the shortest communication time from the source node to the target node.

[0038] The data synchronization module is used to synchronize data with the target node database based on the execution logs of the source node database and the optimal data synchronization scheme.

[0039] On the other hand, embodiments of this specification provide a data synchronization device for a distributed system, wherein the distributed system includes several clusters and an analysis server, each cluster is configured with at least one source node and several target nodes, and the device is applied to the analysis server, including:

[0040] The second receiving module is used to receive the data synchronization order acquisition request sent by the source node;

[0041] The first acquisition module is used to acquire the network status of the source node and the target node according to the data synchronization order;

[0042] The matching module is used to match the most efficient communication transmission method based on the network status.

[0043] The prediction module is used to predict the communication time between nodes under the communication transmission mode, and generate the data synchronization path with the shortest overall communication time based on the communication time.

[0044] The second acquisition module is used to obtain the optimal data synchronization solution based on the data synchronization path.

[0045] The sending module is used to send the optimal data synchronization solution to the source node.

[0046] In another aspect, embodiments of this specification also provide a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the computer program, when executed by the processor, performs instructions of any of the methods described above.

[0047] In another aspect, embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor of a computer device to perform instructions for any of the methods described above.

[0048] Some embodiments of this specification provide one or more technical solutions, which have at least the following technical effects:

[0049] This embodiment first updates the source node database according to the update data request and obtains the execution log of the source node database during the update process. Then, it uses an analysis server to obtain the optimal data synchronization solution with the shortest communication time from the source node to the target node. Finally, it synchronizes the target node database based on the source node's execution log and the optimal data synchronization solution. Therefore, by using the above scheme and the analysis server to obtain the optimal data synchronization solution with the shortest communication time from the source node to each target node, data synchronization efficiency is improved. Furthermore, each target node has independent send and receive capabilities. Once a target node synchronizes data based on the source node, that target node becomes a data source capable of initiating synchronization, reducing the dependence on the source node in the distributed system and avoiding the problem of data synchronization efficiency being affected by source node failure.

[0050] The above description is merely an overview of some embodiments of the technical solutions in this specification. In order to better understand the technical means of some embodiments of this specification and to implement them in accordance with the content of the specification, and to make the above and other objects, features and advantages of the embodiments of this specification more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This specification shows a flowchart illustrating one embodiment of a data synchronization method for a distributed system.

[0053] Figure 2 This specification illustrates a flowchart of a process for determining whether the data synchronization of a target node is abnormal in some embodiments of this specification.

[0054] Figure 3 The flowchart illustrating the monitoring of the target node execution process in some embodiments of this specification is shown;

[0055] Figure 4 This specification shows a schematic diagram of the module structure of a data synchronization device for a distributed system according to one embodiment.

[0056] Figure 5This specification shows a flowchart illustrating another embodiment of a data synchronization method for a distributed system.

[0057] Figure 6 The flowchart shown in some embodiments of this specification illustrates the process of generating the data synchronization path with the shortest overall communication time based on the communication time.

[0058] Figure 7 A schematic diagram illustrating data synchronization using a conventional synchronization sequence is shown.

[0059] Figure 8 This diagram illustrates how data synchronization is performed by generating the optimal data synchronization path using an analytics server.

[0060] Figure 9 This specification shows a schematic diagram of the module structure of another embodiment of a data synchronization device for a distributed system.

[0061] Figure 10 This document illustrates the entire process of data synchronization as described in this manual.

[0062] Figure 11 A schematic diagram of the structure of a computer device is shown in this specification.

[0063] Explanation of symbols in the attached drawings:

[0064] 401. Data Update Module;

[0065] 402. Synchronous Sequence Acquisition Module;

[0066] 403. First receiving module;

[0067] 404, Data Synchronization Module;

[0068] 901. Second receiving module;

[0069] 902. First Acquisition Module;

[0070] 903, Matching Module;

[0071] 904. Prediction Module;

[0072] 905. Second Acquisition Module;

[0073] 906. Sending module;

[0074] 1102. Computer equipment;

[0075] 1104. Processor;

[0076] 1106. Memory;

[0077] 1108. Drive mechanism;

[0078] 1110. Input / output module;

[0079] 1112. Input devices;

[0080] 1114. Output devices;

[0081] 1116. Presentation device;

[0082] 1118. Graphical User Interface;

[0083] 1120. Network interface;

[0084] 1122. Communication link;

[0085] 1124. Communication bus. Detailed Implementation

[0086] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0087] To address the aforementioned problems, this specification provides a data synchronization method for a distributed system. Figure 1 This is a flowchart illustrating a data synchronization method for a distributed system provided in an embodiment of this specification. This specification provides the operational steps of the method described in the embodiment or flowchart, but based on conventional or non-inventive methods, more or fewer operational steps may be included. The order of steps listed in the embodiment is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiment or the accompanying drawings can be executed sequentially or in parallel.

[0088] It should be noted that the terms "first," "second," etc., used in this specification, claims, and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0089] Example 1

[0090] It should be noted that the data synchronization method of the distributed system in this embodiment is applied to the source node side, that is, the execution subject (implementing subject) of the data synchronization method of the distributed system in the following embodiments of this specification is the source node side.

[0091] Reference Figure 1 As shown in the embodiments of this specification, a data synchronization method for a distributed system is provided. The distributed system includes several clusters and an analysis server. Each cluster is configured with at least one source node and several target nodes. The method is applied to the source nodes and includes:

[0092] S101: Receive an update data request, update the source node database according to the update data request, and obtain the execution log of the source node database;

[0093] S102: Send a data synchronization sequence acquisition request to the analysis server;

[0094] S103: Receive the optimal data synchronization scheme sent by the analysis server, wherein the optimal data synchronization scheme includes the data synchronization path with the shortest communication time from the source node to the target node;

[0095] S104: Synchronize the target node database according to the execution log of the source node database and the optimal data synchronization scheme.

[0096] This embodiment first updates the source node database according to the update data request and obtains the execution log of the source node database during the update process. Then, it uses an analysis server to obtain the optimal data synchronization solution with the shortest communication time from the source node to the target node. Finally, it synchronizes the target node database based on the source node's execution log and the optimal data synchronization solution. Therefore, by using the above scheme and the analysis server to obtain the optimal data synchronization solution with the shortest communication time from the source node to each target node, data synchronization efficiency is improved. Furthermore, each target node has independent send and receive capabilities. Once a target node synchronizes data based on the source node, that target node becomes a data source capable of initiating synchronization, reducing the dependence on the source node in the distributed system and avoiding the problem of data synchronization efficiency being affected by source node failure.

[0097] In this specification, the distributed system comprises several clusters and an analysis server. Different clusters implement different business processes, and the analysis server generates the optimal data synchronization solution with the shortest communication time between clusters. Each cluster can be a collection of multiple computer programs that utilize computing resources across multiple independent computing nodes to implement the same business process. These programs communicate and synchronize via a public network using different nodes. These nodes are typically independent physical hardware devices, such as servers, but can also be independent software processes or other recursively encapsulated systems. Each cluster contains at least one source node and several target nodes. The target nodes are the servers that need to synchronize data, i.e., they synchronize data based on the data updates from the source nodes. Data from each node in the cluster is stored in a corresponding database, which can be a relational database such as MySQL, Oracle, or DB2. The data warehouse corresponding to the analysis server is a Hive data warehouse, used to store log data generated during business processing by the analysis server.

[0098] When data is updated in the source node's database, data synchronization is required for the remaining nodes in the same cluster. Specifically, the source node first receives an external update request, updates the corresponding database based on the request, and retrieves the execution log for that database. Since the source node does not know the optimal synchronization path to the target node, it uses an analytics server to obtain this optimal path. Then, it initiates data synchronization with the target node based on this optimal path, which includes the shortest communication time between the source and target nodes. The source node's database execution log includes the database binary log, log date, log version number, and log file location. The source node's binary log records all the operation instructions for updating data in the source node's database. These instructions can be copied to the target node's database and re-executed, thus achieving data synchronization. The log date and version number prevent duplicate synchronization operations, preventing system loops that could affect data synchronization efficiency. The log file location facilitates locating the corresponding binary log storage location, speeding up log file lookup.

[0099] Furthermore, data synchronization is performed on the target node database based on the execution logs of the source node database and the optimal data synchronization scheme. This includes: adding the execution logs of the source node database to the relay logs of the target node database sequentially according to the data synchronization order of the target nodes; and updating each target node database based on the events in the relay logs. The data synchronization order is the data synchronization execution order, which is generated by the analysis server. Synchronizing the target nodes according to this order minimizes communication time, and each target node is a send-receiver server during data synchronization. When source node A synchronizes data to target node B, source node A and target node B become two data sources of equal level capable of initiating synchronization. Target node B can then initiate data synchronization to other target nodes, avoiding excessive dependence on the source node and improving data synchronization efficiency. The relay log is an intermediate file used by the target node to store the binary log content of the source node database. It is the core connecting the source and target nodes, acting as a buffer, so that the source node does not have to wait for the target node to complete execution before sending the next binary log. Specifically, the execution log of the source node database includes a binary log, which records the set of operation instructions for updating all data in the source node database. For example, if the source node database is a MySQL database, the operation instructions are SQL statements, namely the database's insert, delete, and update operation statements. This binary log is copied to the relay log of the target node database, and the target node database performs corresponding operations on its database based on the SQL statements therein, thereby achieving data synchronization.

[0100] In the embodiments of this specification, to avoid abnormal situations during data synchronization that could lead to data synchronization failures at the target node, refer to... Figure 2 The method further includes:

[0101] S201: Obtain the updated data from the source node database and the target node database;

[0102] S202: Compare the updated data in the target node database with the updated data in the source node database to obtain a comparison result;

[0103] S203: Determine whether the data synchronization of the target node is abnormal based on the comparison results;

[0104] S204: If so, an alarm notification is sent to the target node where the data synchronization error occurred, and the data synchronization of the target node where the data synchronization error occurred is re-performed.

[0105] Therefore, by following the above steps, the data content of the source node and the target node can be compared. If the comparison results are inconsistent, it indicates that the data synchronization of the target node database has failed. In this case, the target node database that has failed to synchronize needs to be resynchronized to ensure the accuracy of data synchronization.

[0106] In the embodiments of this specification, to avoid situations where the relay log of the target node database encounters anomalies during the execution of statements in the binary log, thereby causing data synchronization failure, refer to... Figure 3 The method further includes:

[0107] S301: When the database of each target node is updated according to the events in the relay log, the execution process of the target node is monitored to obtain the execution status of the target node;

[0108] S302: If the execution status of the target node is failed and the number of failures is less than the preset number, then the target node is resynchronized.

[0109] Therefore, by monitoring the status of statements in the relay log of the target node database through the above steps, the target node that fails to execute can be detected in a timely manner, and its data can be resynchronized. This avoids the situation where data synchronization fails due to abnormalities in the statements in the binary log of the target node database's relay log, thereby improving data synchronization efficiency.

[0110] Based on the data synchronization method for a distributed system described above, this specification also provides a corresponding apparatus for a data synchronization method for a distributed system. The apparatus may include a system (including a distributed system), software (application), module, component, server, client, etc., using the method described in the embodiments of this specification, combined with necessary hardware implementation. Based on the same innovative concept, the apparatuses in one or more embodiments provided in this specification are as described in the following embodiments. Since the implementation schemes and methods for solving the problem by the apparatus are similar, the implementation of the specific apparatus in the embodiments of this specification can refer to the implementation of the aforementioned method, and repeated details will not be repeated. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0111] Specifically, Figure 4 This is a schematic diagram of the module structure of one embodiment of a data synchronization device for a distributed system provided in this specification. (Refer to...) Figure 4 As shown in the embodiments of this specification, a data synchronization device for a distributed system includes:

[0112] The data update module 401 is used to receive update data requests, update the source node database according to the update data requests, and obtain the execution log of the source node database;

[0113] Synchronization order acquisition module 402 is used to send a data synchronization order acquisition request to the analysis server;

[0114] The first receiving module 403 is used to receive the optimal data synchronization scheme sent by the analysis server, wherein the optimal data synchronization scheme includes the data synchronization path with the shortest communication time from the source node to the target node.

[0115] The data synchronization module 404 is used to synchronize data with the target node database based on the execution log of the source node database and the optimal data synchronization scheme.

[0116] The beneficial effects obtained by the apparatus provided in the embodiments of this specification are consistent with the beneficial effects obtained by the methods described above, and will not be repeated here.

[0117] Example 2

[0118] It should be noted that the data synchronization method of the distributed system in this embodiment is applied to the analysis server side, that is, the execution subject (implementation subject) of the data synchronization method of the distributed system in the following embodiments of the present invention is the analysis server.

[0119] Reference Figure 5 As shown in some embodiments of this specification, a data synchronization method for a distributed system includes the following steps:

[0120] S501: Receive the data synchronization order acquisition request sent by the source node;

[0121] S502: Obtain the network status of the source node and the target node according to the data synchronization order;

[0122] S503: Match the most efficient communication transmission method based on the network status;

[0123] S504: Predict the communication time between nodes under the aforementioned communication transmission method, and generate the data synchronization path with the shortest overall communication time based on the communication time.

[0124] S505: Based on the data synchronization path, obtain the optimal data synchronization solution;

[0125] S506: Send the optimal data synchronization solution to the source node.

[0126] The source and target nodes communicate and synchronize via a public network. The data synchronization request includes the proxy addresses of each node. These proxy addresses mask the real network address information, providing a secure, stable, and anonymous network connection, while also verifying legitimate information and filtering illegal or abnormal data. The network status of each node can be obtained through the proxy addresses. The analysis server matches the most efficient communication transmission method based on the network status of each node. Available communication transmission methods include RPC, TCP, file, Kafka, and user-defined APIs. By utilizing the most efficient communication transmission method for communication and synchronization between nodes, data synchronization is ensured to proceed with the fastest communication, thereby improving data synchronization efficiency. After selecting the most efficient communication transmission method, the analysis server predicts the communication time between nodes under this method. Based on the communication time, the data synchronization path with the shortest overall time is generated and sent to the source node as the optimal data synchronization solution.

[0127] Furthermore, in the embodiments of this specification, reference is made to... Figure 6 The process of generating the data synchronization path with the shortest overall communication time based on the communication time includes the following steps:

[0128] S1: Starting from the source node;

[0129] S2: Find the target node with the shortest communication time with the source node as the second node, and use the second node as the new starting point to find the target node with the shortest communication time with the second node as the third node;

[0130] S3: Calculate the communication time of the source node through the second node to the third node and calculate the communication time of the source node to the third node;

[0131] S4: Select the path with the shortest communication time as the shortest path from the starting point to the third node;

[0132] S5: Using the third node as the new starting point, repeat steps S2-S4 until the shortest path from the source node to all target nodes is determined.

[0133] Reference Figure 7 As shown, assuming a cluster contains one source node A and four target nodes B, C, D, and E, a total of five nodes, the analysis server is first used to predict the communication time t between the nodes under the communication transmission method with the highest efficiency. ij , t ij Let t represent the communication time from node i to node j. If synchronization is performed using the conventional synchronization sequence, the total time is the sum of the times taken from source node A to each target node, i.e., t = t_j. AB +tAC +t AD +t AE This synchronization sequence initiates synchronization from source node A to each target node sequentially. This process is time-consuming, inefficient, and relies solely on source node A. If source node A malfunctions, it affects the overall data synchronization progress, resulting in low efficiency. (Referring to...) Figure 8 As shown, this application uses an analysis server to find the shortest synchronization path. Specifically, it assumes that the communication time between each node is t. AB =9,t AE =3, t AC =8, t AD =12, t BC =7,t BD =5,t BE =3, t CD =5,t CE =5,t DE =5. Starting from source node A, find the target node with the shortest communication time, i.e., target node E. Therefore, the shortest path from source node A to target node E is: source node A → target node E. Taking target node E as the new starting point, find the target node with the shortest communication time with target node E, i.e., target node B. At this time, a path is generated: source node A → target node E → target node B. Calculate the communication time of this path as: t′ AB =t AE +t BE =6, t′ AB With t AB By comparison, we can know t′ AB <t AB Therefore, the shortest path from source node A to target node B is source node A → target node E → target node B. Taking target node B as the new starting point, we find the target node with the shortest communication time, which is target node D. At this point, a path is generated: source node A → target node E → target node B → target node D. The communication time of this path is calculated as: t′ AD =t AE +t BE +t BD =11, t′ AD With t AD By comparison, we can know t′ AD <t AD Therefore, the shortest path from source node A to target node D is source node A → target node E → target node B → target node D. Now, only the target node C remains. The communication time for source node A → target node E → target node B → target node D → target node C is t′. AC =t AE +t BE +tBD +t CD =16, t′ AC With t AC By comparison, we can know t′ AD >t AD Therefore, the shortest path from source node A to target node C is source node A → target node C. Thus, the shortest paths from the source node to all target nodes can be determined, and the total computation time is t′ = t. AE +t BE +t BD +t AC =19, compared to the conventional synchronous sequential method, the total time is t=t AB +t AC +t AD +t AE =32, the synchronization path obtained using this application takes significantly less time, thereby improving data synchronization efficiency.

[0134] Based on the data synchronization method for a distributed system described above, this specification also provides a corresponding data synchronization device for a distributed system. Specifically, Figure 9 This is a schematic diagram of the module structure of one embodiment of a data synchronization device for a distributed system provided in this specification. (Refer to...) Figure 9 As shown in the embodiments of this specification, a data synchronization device for a distributed system includes:

[0135] The second receiving module 901 is used to receive the data synchronization order acquisition request sent by the source node;

[0136] The first acquisition module 902 is used to acquire the network status of the source node and the target node according to the data synchronization order;

[0137] Matching module 903 is used to match the most efficient communication transmission method according to the network status;

[0138] Prediction module 904 is used to predict the communication time between nodes under the communication transmission mode, and generate the data synchronization path with the shortest overall communication time based on the communication time.

[0139] The second acquisition module 905 is used to obtain the optimal data synchronization solution based on the data synchronization path;

[0140] The sending module 906 is used to send the optimal data synchronization scheme to the source node.

[0141] The beneficial effects obtained by the apparatus provided in the embodiments of this specification are consistent with the beneficial effects obtained by the methods described above, and will not be repeated here.

[0142] Example 3

[0143] This embodiment describes the entire process of data synchronization in a distributed system. (Refer to...) Figure 10 As shown, the source node receives an update data request, updates its database accordingly, and returns a success message. Next, the source node sends a request to its database to retrieve the execution log. Upon receiving this request, the source node returns the execution log to its database. After receiving the execution log, the source node prepares to initiate data synchronization with the target nodes. To improve data synchronization efficiency, the source node sends a data synchronization order retrieval request to the analysis server. Upon receiving this request, the analysis server generates an optimal data synchronization plan and sends it to the source node. The source node then synchronizes data with the remaining target nodes based on the optimal plan and the execution log. Thus, this approach utilizes the analysis server to obtain the optimal data synchronization plan with the shortest communication time from the source node to each target node, improving data synchronization efficiency. Furthermore, each target node has independent send and receive capabilities. Once a target node synchronizes data with the source node, that target node becomes a data source capable of initiating synchronization, reducing the dependence on the source node in the distributed system and preventing data synchronization efficiency from being affected by source node failures.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the acquisition, storage, use, and processing of data in the technical solutions described in the embodiments of this application all comply with relevant regulations.

[0145] It should be noted that the data synchronization method and apparatus for distributed systems described in this specification can be used in the financial field, or in any field other than the financial field. The application fields of the data synchronization method and apparatus for distributed systems described in this specification are not limited.

[0146] Reference Figure 11As shown, based on the data synchronization method for a distributed system described above, one embodiment of this specification also provides a computer device 1102, wherein the above method operates on the computer device 1102. The computer device 1102 may include one or more processors 1104, such as one or more central processing units (CPUs), each of which can implement one or more hardware threads. The computer device 1102 may also include any memory 1106 for storing information of any kind, such as code, settings, data, etc. Non-limitingly, for example, the memory 1106 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 1102. In one case, when the processor 1104 executes associated instructions stored in any memory or combination of memories, the computer device 1102 can perform any operation of the associated instructions. The computer device 1102 also includes one or more drive mechanisms 1108 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0147] Computer device 1102 may further include an input / output module 1110 (I / O) for receiving various inputs (via input device 1112) and providing various outputs (via output device 1114). A specific output mechanism may include a presentation device 1116 and an associated graphical user interface (GUI) 1118. In other embodiments, the input / output module 1110 (I / O), input device 1112, and output device 1114 may be omitted, and the device may function solely as a computer device within a network. Computer device 1102 may also include one or more network interfaces 1120 for exchanging data with other devices via one or more communication links 1122. One or more communication buses 1124 couple the components described above together.

[0148] Communication link 1122 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 1122 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0149] Corresponding to, for example Figures 1 to 3 and Figures 5 to 8 and Figure 10In addition to the method shown, embodiments of this specification also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the above-described method.

[0150] This specification also provides computer-readable instructions, wherein when a processor executes the instructions, the program therein causes the processor to perform the following... Figures 1 to 3 and Figures 5 to 8 and Figure 10 The method.

[0151] This specification also provides a computer program product, including at least one instruction or at least one program segment, wherein the at least one instruction or the at least one program segment is loaded and executed by a processor to achieve the following: Figures 1 to 3 and Figures 5 to 8 and Figure 10 The method.

[0152] It should be understood that in the various embodiments of this specification, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0153] It should also be understood that, in the embodiments of this specification, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this specification generally indicates that the preceding and following related objects have an "or" relationship.

[0154] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this specification.

[0155] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0156] In the several embodiments provided in this specification, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0157] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described in this specification, depending on actual needs.

[0158] Furthermore, the functional units in the various embodiments of this specification can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0159] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this specification. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] This specification uses specific embodiments to illustrate the principles and implementation methods of this specification. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this specification. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this specification. Therefore, the content of this specification should not be construed as a limitation of this specification.

Claims

1. A data synchronization method for a distributed system, characterized in that, The distributed system includes several clusters and an analysis server. Each cluster is configured with at least one source node and several target nodes. The method is applied to the source nodes and includes: Receive an update data request, update the source node database according to the update data request, and obtain the execution log of the source node database; Send a data synchronization sequence acquisition request to the analysis server; The system receives the optimal data synchronization scheme sent by the analysis server, wherein the optimal data synchronization scheme includes the data synchronization path with the shortest communication time from the source node to the target node. Data synchronization is performed on the target node database based on the execution logs of the source node database and the optimal data synchronization scheme.

2. The method according to claim 1, characterized in that, The step of synchronizing data to the target node database based on the execution logs of the source node database and the optimal data synchronization scheme includes: The execution logs of the source node database are added to the relay logs of the target node database in the order of data synchronization of the target nodes. Each target node database is updated according to the events in the relay logs.

3. The method according to claim 1, characterized in that, The method further includes: Obtain the updated data from the source node database and the target node database; The updated data in the target node database is compared with the updated data in the source node database to obtain the comparison result; Based on the comparison results, determine whether the data synchronization of the target node is abnormal; If so, an alarm notification is sent to the target node where the data synchronization error occurred, and the data synchronization of the target node where the data synchronization error occurred is re-performed.

4. The method according to claim 2, characterized in that, The method further includes: When the database of each target node is updated according to the events in the relay log, the execution process of the target node is monitored to obtain the execution status of the target node; If the execution status of the target node is failed and the number of failures is less than the preset number, then the target node will be resynchronized.

5. A data synchronization method for a distributed system, characterized in that, The distributed system comprises several clusters and an analysis server, each cluster being configured with at least one source node and several target nodes. The method is applied to the analysis server and includes: Receive the data synchronization order acquisition request sent by the source node; According to the data synchronization order, the network status of the source node and the target node is obtained by requesting; The most efficient communication transmission method is matched based on the network status. Predict the communication time between nodes under the aforementioned communication transmission method, and generate the data synchronization path with the shortest overall communication time based on the communication time. Based on the data synchronization path, the optimal data synchronization solution is obtained; Send the optimal data synchronization solution to the source node.

6. The method according to claim 5, characterized in that, The step of generating the data synchronization path with the shortest overall communication time based on the communication time includes: S1: Starting from the source node; S2: Find the target node with the shortest communication time with the source node as the second node, and use the second node as the new starting point to find the target node with the shortest communication time with the second node as the third node; S3: Calculate the communication time of the source node through the second node to the third node and calculate the communication time of the source node to the third node; S4: Select the path with the shortest communication time as the shortest path from the starting point to the third node; S5: Using the third node as the new starting point, repeat steps S2-S4 until the shortest path from the source node to all target nodes is determined.

7. A data synchronization device for a distributed system, characterized in that, The distributed system includes several clusters and an analysis server. Each cluster is configured with at least one source node and several target nodes. The device is applied to the source node and includes: The data update module is used to receive update data requests, update the source node database according to the update data requests, and obtain the execution log of the source node database; The synchronization order acquisition module is used to send a data synchronization order acquisition request to the analysis server; The first receiving module is used to receive the optimal data synchronization scheme sent by the analysis server, wherein the optimal data synchronization scheme includes the data synchronization path with the shortest communication time from the source node to the target node. The data synchronization module is used to synchronize data with the target node database based on the execution logs of the source node database and the optimal data synchronization scheme.

8. A data synchronization device for a distributed system, characterized in that, The distributed system includes several clusters and an analysis server. Each cluster is configured with at least one source node and several target nodes. The device is applied to the analysis server and includes: The second receiving module is used to receive the data synchronization order acquisition request sent by the source node; The first acquisition module is used to acquire the network status of the source node and the target node according to the data synchronization order; The matching module is used to match the most efficient communication transmission method based on the network status. The prediction module is used to predict the communication time between nodes under the communication transmission mode, and generate the data synchronization path with the shortest overall communication time based on the communication time. The second acquisition module is used to obtain the optimal data synchronization solution based on the data synchronization path. The sending module is used to send the optimal data synchronization solution to the source node.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images