A method, apparatus and system for implementing data synchronization

Abstract

The application discloses a method, device and system for realizing data synchronization. Since the synchronization task notifies the identification information of the synchronization task itself when notifying a remote cluster to delete the log record corresponding to a point, the subsequent deletion of the source cluster can be performed on the log record corresponding to the minimum point. In this way, the loss of the synchronized data is avoided, the common storage for saving the point information applied by different synchronization tasks does not need to be introduced and maintained, the complexity of the system is reduced, and the maintenance cost is greatly reduced.

Description

Technical Field

[0001] This application relates to, but is not limited to, database technology, and in particular to a method, apparatus and system for achieving data synchronization. Background Technology

[0002] Data analysis, especially real-time data analysis, has increasingly become a requirement and standard across various industries. binlog, short for Binary Log, is used for master-slave backup or data recovery. Specifically, records of database modifications are stored in binlog files. This allows for data recovery after a database failure, and also enables the replay of binlogs generated by the source database on the destination database for data synchronization. Each database modification record in the binlog is assigned a sequence number, which increments from smallest to largest. This sequence number is also called a bit position. Smaller bits represent older writes, while larger bits represent more recent writes.

[0003] Figure 1 This is a schematic diagram illustrating an embodiment of data synchronization in the related art, where data is synchronized from a source cluster to a destination cluster. Figure 1 As shown, assuming cluster A is the source cluster and cluster B is the destination cluster, during data synchronization, a synchronization task R is used to pull binlog from cluster A and send it to cluster B. A synchronization task is a software task; that is, a synchronization task can be added to an operations and maintenance management system. This synchronization task is responsible for synchronizing data from the source cluster to the destination cluster. Different synchronization tasks have unique identification information. To ensure data continuity and prevent loss, synchronization task R obtains the binlog corresponding to point 1, 2, 3 from cluster A (e.g., ...). Figure 1 Step 1) has been completed and the binlog from point 111 has been applied to cluster B (e.g., ...). Figure 1 In step 2), the synchronization task R will notify cluster A to delete the binlog before point 111 (e.g., ...). Figure 1 (Step 3 in the process). If synchronization task R does not notify cluster A, then cluster A will not delete binlogs in the future. This will cause the binlog files in cluster A to accumulate and eventually fill up the disk. If cluster A deletes binlogs on its own or prematurely instead of following the instructions of synchronization task R, then the binlog corresponding to point 124 may be deleted by cluster A before synchronization task R has even received the record corresponding to point 124. This will result in discontinuous data synchronization and data loss for synchronization task R. Summary of the Invention

[0004] This application provides a method, apparatus, and system for achieving data synchronization, which can reduce the complexity of synchronization tasks and improve their robustness.

[0005] This invention provides a method for achieving data synchronization, comprising:

[0006] The synchronization task pulls log records from the source cluster and applies them to the destination cluster corresponding to the synchronization task.

[0007] The synchronization task notifies the source cluster of the log records it pulls, including the location information of the log records applied in the destination cluster corresponding to the synchronization task, and the identification information of the synchronization task.

[0008] In one exemplary instance, the same synchronization task corresponds to one destination cluster, and different synchronization tasks correspond to different destination clusters.

[0009] This application provides a computer-readable storage medium storing computer-executable instructions for performing any of the above-described methods for achieving data synchronization.

[0010] This application provides an apparatus for implementing data synchronization, including a memory and a processor, wherein the memory stores the following instructions executable by the processor for performing the steps of the data synchronization method described in any of the preceding claims.

[0011] This application also provides a method for achieving data synchronization, including:

[0012] The source cluster receives a notification from the synchronization task, which includes the point information of the pulled log records that have been applied in the destination cluster corresponding to the synchronization task and the identification information of the synchronization task.

[0013] The source cluster determines the minimum location information from the location information notified by different synchronization tasks based on the identification information of the synchronization tasks.

[0014] The source cluster deletes log records corresponding to points smaller than the determined minimum point.

[0015] In one exemplary instance, the method further includes:

[0016] The source cluster receives a notification from the synchronization task and records the time when the synchronization task last updated and deleted the point.

[0017] If the synchronization task is not updated within the preset time period, delete the identification information of the synchronization task and the location information of the synchronization task notification.

[0018] In one exemplary instance, the same synchronization task corresponds to a destination cluster.

[0019] This application provides a computer-readable storage medium storing computer-executable instructions for executing the source-end cluster data synchronization method described in any of the above claims.

[0020] This application provides an apparatus for implementing data synchronization, including a memory and a processor, wherein the memory stores the following instructions executable by the processor: steps for performing the source-end cluster data synchronization method described in any of the preceding claims.

[0021] This application also provides a system for implementing data synchronization, comprising: a source cluster, multiple synchronization tasks, and multiple destination clusters, wherein...

[0022] The synchronization task is used to pull log records from the source cluster and apply them to the destination cluster corresponding to the synchronization task; it notifies the source cluster of the location information of the pulled log records applied in the destination cluster corresponding to the synchronization task and the identification information of the synchronization task.

[0023] The source cluster is used to receive notifications from synchronization tasks. The notifications include the location information of the log records that have been applied in the destination cluster corresponding to the synchronization task and the identification information of the synchronization task. Based on the identification information of the synchronization task, the minimum location information is determined from the location information of different synchronization task notifications. Log records corresponding to locations less than the determined minimum location are deleted.

[0024] The destination cluster applies log records pulled from the corresponding synchronization task and returns the applied point information to the corresponding synchronization task.

[0025] In one exemplary instance, the source cluster is further used for:

[0026] Upon receiving a notification from the synchronization task, the time of the last update and deletion of the points by the synchronization task is recorded; if the synchronization task does not perform any further updates within a preset time period, the identification information of the synchronization task and the point information notified by the synchronization task are deleted.

[0027] In this embodiment, when the synchronization task notifies the remote cluster to delete the log record corresponding to the point, it also notifies the synchronization task's own identification information. In this way, the source cluster can distinguish between multiple synchronization tasks that have issued point information for deleting log records, and find the smallest point among the point information issued by these synchronization tasks. This means that the source cluster will subsequently delete the log record corresponding to the smallest point. This not only avoids the loss of synchronization data, but also eliminates the need to introduce and maintain shared storage to save the point information applied by different synchronization tasks, reducing the complexity of the system and greatly reducing maintenance costs.

[0028] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0029] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0030] Figure 1 This is a schematic diagram illustrating an embodiment of data synchronization in the related art, where data is synchronized from a source cluster to a destination cluster.

[0031] Figure 2 This is a schematic diagram illustrating another embodiment of data synchronization in the related art, where data is synchronized from a source cluster to multiple destination clusters.

[0032] Figure 3 This is a schematic diagram illustrating yet another embodiment of data synchronization in the context of synchronizing data from a source cluster to multiple destination clusters in the related art;

[0033] Figure 4 This is a flowchart illustrating a method for achieving data synchronization according to this application;

[0034] Figure 5 A schematic diagram illustrating an embodiment of data synchronization implemented in this application;

[0035] Figure 6 This is a flowchart illustrating another method for achieving data synchronization according to this application;

[0036] Figure 7 This is a schematic diagram of the system architecture for implementing data synchronization in this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be arbitrarily combined with each other.

[0038] In a typical configuration of this application, the computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0039] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0040] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0041] The steps illustrated in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases the steps shown or described may be performed in a different order than that shown here.

[0042] If data synchronization involves synchronizing from one source cluster to multiple destination clusters, the problem becomes more complex.

[0043] In related technologies, one approach is to pull binlog from the source cluster within the same synchronization task and distribute it to each destination cluster. The advantage of this is that the synchronization task itself knows the minimum point at which data has been applied to the destination clusters and can correctly notify the source cluster to delete the records corresponding to that minimum point. For example... Figure 2 As shown, assuming cluster A is the source cluster, and clusters B, C, and D are the destination clusters, during data synchronization, a synchronization task R pulls binlog from cluster A, processes it, and distributes it to clusters B, C, and D. In this embodiment, it is assumed that synchronization task R obtains the binlog corresponding to point 333 from cluster A (e.g., ...). Figure 2 Step 1) has been completed and the binlog from point 122 has been applied to cluster B (e.g., ...). Figure 2 Step 2) has applied the binlog from point 333 and earlier to cluster C (e.g., ...). Figure 2Step 3) has applied the binlog from point 67 and earlier to cluster D (e.g., ...). Figure 2 In step 4), the synchronization task R will notify cluster A to delete the binlog before the minimum point 67 (e.g., ...). Figure 2 (Step 5 in the process). However, the inventors of this application have discovered that, on the one hand, the scale, processing capacity, and network conditions of clusters B, C, and D may be different; on the other hand, the speed at which clusters B, C, and D apply binlog may also vary greatly. This can lead to complex management issues in the same synchronization task R. Moreover, if synchronization task R fails, since multiple destination clusters share a single synchronization task, the three synchronization tasks to clusters B, C, and D will all be affected, thus expanding the scope of the failure.

[0044] like Figure 3 As shown, assuming cluster A is the source cluster and clusters B, C and D are the destination clusters, another approach in related technologies is to start their own synchronization tasks, such as synchronization task R(B), synchronization task R(C) and synchronization task R(D), for different destination clusters. All three synchronization tasks pull binlog from cluster A and synchronize it to the corresponding clusters B, C and D. In this scenario, if each of the three synchronization tasks sends the minimum point information it knows about that has been applied to the target cluster to cluster A, and synchronization task R(B) sends point 122, synchronization task R(C) sends point 333, and synchronization task R(D) sends point 67, cluster A will perform record deletion processing according to the received point information. For example, if cluster A receives point 122 from R(B), it will delete the binlog before point 122. However, the inventors of this application discovered that since synchronization task R(D) has only processed point 67 at this point, it will be unable to obtain continuous binlog, resulting in the loss of synchronization data. In this case, a shared storage S can be introduced to assist in determining the minimum point information. In the shared storage S, synchronization tasks R(B), R(C), and R(D) respectively write their own point information that has been applied to clusters B, C, and D (e.g., ...). Figure 3 In steps 13, 23, and 33, the public storage S or other tasks, such as one or more synchronization tasks, calculate the minimum deletable binlog point (i.e., the minimum point) and then notify cluster A (e.g., ...). Figure 3 Step 4) in the process allows cluster A to properly delete binlog. While introducing shared storage S decouples multiple synchronization tasks, it also increases system complexity and maintenance costs due to the need to introduce and maintain the shared storage itself.

[0045] To improve the robustness and reduce the complexity of synchronization tasks, this application provides a method for implementing data synchronization. Figure 4 This is a flowchart illustrating a method for implementing data synchronization according to this application, as shown below. Figure 4 As shown, it includes:

[0046] Step 400: The synchronization task pulls log records from the source cluster and applies them to the destination cluster corresponding to the synchronization task.

[0047] In one exemplary instance, the synchronization task includes two or more tasks.

[0048] In one exemplary instance, the same synchronization task corresponds to a single destination cluster.

[0049] In one exemplary instance, a common implementation of logging may include binlog.

[0050] Step 401: The synchronization task notifies the source cluster of the location information of the log records pulled and the identification information of the synchronization task in the destination cluster corresponding to the synchronization task.

[0051] In one exemplary instance, the identification information for each synchronization task is a unique identifier, such as X1, X2, X3, etc. The identification information for synchronization tasks can be assigned at startup.

[0052] In practical applications, synchronization tasks can be initiated by the user through an operation center (or operations and maintenance platform). The task assigned to the synchronization task is to synchronize data from a specified cluster at the source end to a specified cluster at the destination end. The operation center (or operations and maintenance platform) initiating the synchronization task is responsible for assigning identification information to the task. The operations and maintenance platform can record and manage the synchronization tasks and their identification information. Alternatively, the synchronization task itself can generate a unique identification information, which will be stored for recovery if the task crashes. The identification information of the synchronization task can include universally unique identifiers (UUIDs) or globally unique identifiers (GUIDs).

[0053] In this application, since the synchronization task notifies the source cluster of its own identification information when it notifies the source cluster to delete the log record corresponding to the point, the source cluster can distinguish between multiple synchronization tasks that have issued point information for deleting log records, and find the smallest point among the point information issued by these synchronization tasks. This means that the source cluster will subsequently delete the log record corresponding to the smallest point. In this way, not only is the loss of synchronization data avoided, but there is also no need to introduce and maintain public storage, which reduces the complexity of the system and greatly reduces maintenance costs.

[0054] This application also provides a computer-readable storage medium storing computer-executable instructions, the computer-executable instructions being used to perform... Figure 4 The method for implementing data synchronization for any of the items shown.

[0055] This application further provides an apparatus for implementing data synchronization, including a memory and a processor, wherein the memory stores the following instructions executable by the processor: for executing Figure 4 The steps of the method for achieving data synchronization as described in any of the examples shown.

[0056] This application also provides a method for achieving data synchronization. Figure 6 This is a flowchart illustrating another method for achieving data synchronization according to this application, as shown below. Figure 6 As shown, it includes:

[0057] Step 600: The source cluster receives a notification from the synchronization task, which includes the location information of the pulled log records that have been applied in the destination cluster corresponding to the synchronization task and the identification information of the synchronization task.

[0058] In one exemplary instance, the synchronization task includes two or more tasks.

[0059] In one exemplary instance, the same synchronization task corresponds to one destination cluster, and different synchronization tasks correspond to different destination clusters.

[0060] In one exemplary instance, the identification information for each synchronization task is a unique identifier, such as X1, X2, X3, etc. The identification information for synchronization tasks can be assigned at startup.

[0061] It should be noted that the source cluster is a software cluster (or application cluster). That is, the software application is deployed on a group of machines and provides services to the outside world. From the outside, it appears to be a server cluster that provides the same service.

[0062] Step 601: The source cluster determines the minimum location information from the location information notified by different synchronization tasks based on the identification information of the synchronization task.

[0063] In one exemplary instance, the source cluster can periodically execute step 601. That is, it periodically finds the minimum location information by checking all synchronization task identifiers and corresponding location information received by the remote cluster itself.

[0064] Figure 5 This is a schematic diagram illustrating an embodiment of data synchronization implemented in this application, as shown below. Figure 5 As shown, assume cluster A is the source cluster, and clusters B, C, and D are the destination clusters. Synchronization task R(B) corresponds to destination cluster B, synchronization task R(C) corresponds to destination cluster C, and synchronization task R(D) corresponds to destination cluster D. For example, if synchronization task R(B) first notifies cluster A to delete the binlog corresponding to point 122 (e.g., ... Figure 5 Step 13), that is, the point that the synchronization task R(B) itself has applied is point 122 (that is, it has applied the binlog before point 122 to cluster B), and then the synchronization task R(C) notifies cluster A to delete the binlog corresponding to point 333 (e.g. Figure 5 In step 23), the synchronization task R(C) itself has already applied point 333 (that is, it has applied the binlog before point 333 to cluster C). Then, the synchronization task R(D) notifies cluster A to delete the binlog corresponding to point 67 (e.g., ...). Figure 5 In step 33), the point that the synchronization task R(D) itself has applied is point 67 (that is, the binlog before point 67 that was pulled has been applied to cluster D); cluster A can know the minimum allowed deletion point is point 67 notified by synchronization task R(D) based on the different identifier information of synchronization tasks R(B), R(C), and R(D), such as X1, X2, and X3 (e.g., point 67). Figure 5 As per step 4), cluster A will only delete binlogs before point 67, and will not delete binlogs before point 333 or point 122. This decouples multiple synchronization tasks without introducing additional public storage, which not only avoids the loss of synchronization data, but also eliminates the need to introduce and maintain public storage, reducing system complexity and significantly reducing maintenance costs.

[0065] Step 602: Delete the log records corresponding to points smaller than the determined minimum point from the source cluster.

[0066] The data synchronization method provided in this application introduces a unique identifier for each synchronization task and includes this identifier when notifying the source cluster to delete binlog. In this way, by maintaining the deletion point information of multiple identifiers in the source cluster, multiple synchronization tasks can be used to synchronize data from the same cluster to different destination clusters independently and without each other's awareness. This reduces the complexity of synchronization tasks, improves their robustness, and avoids the introduction of a common storage, which greatly reduces the construction and maintenance costs of the synchronization system.

[0067] In one exemplary instance, this application may further include:

[0068] When the source cluster receives a notification from the synchronization task, it records the time when the synchronization task last updated and deleted the data point. If the synchronization task does not perform any updates within a preset time period, the source cluster automatically deletes the identifier information of the synchronization task and the data point information notified by the synchronization task. It should be noted that the source cluster records the time information of receiving each notification.

[0069] In one exemplary instance, the destination cluster applies the log records pulled from the corresponding synchronization task and returns the applied location information to the corresponding synchronization task.

[0070] Thus, combined Figure 5 As shown, if the synchronization task R(B) issues a notification to delete to point 122 (e.g.) Figure 5 If cluster A crashes after step 13), because cluster A records the time when the corresponding synchronization task last updated the deleted point information (i.e., the time when it last sent a notification to cluster A) when it receives the deleted point information and identification information from each synchronization task, then if cluster A does not receive any further notification from that synchronization task to update the deleted point information within a preset time period, such as 6 hours (e.g., ...), Figure 5 In step 5), cluster A will directly delete the identifier information of the synchronization task corresponding to the identifier information and the previously sent deletion point information. This effectively avoids the situation where the binlog can only be deleted up to point 122 in the future.

[0071] This application also provides a computer-readable storage medium storing computer-executable instructions, the computer-executable instructions being used to perform... Figure 6 The method for implementing data synchronization for any of the items shown.

[0072] This application further provides an apparatus for implementing data synchronization, including a memory and a processor, wherein the memory stores the following instructions executable by the processor: for executing Figure 6The steps of the method for achieving data synchronization as described in any of the examples shown.

[0073] Figure 7 This is a schematic diagram of the system architecture for implementing data synchronization in this application, as shown below. Figure 7 As shown, this includes the source cluster and multiple synchronization tasks (such as...). Figure 7 Synchronization tasks R(1), R(2), ..., R(n) in the middle, and multiple destination clusters (such as... Figure 7 In the context of destination clusters 1, 2, ..., n), where,

[0074] The synchronization task is used to pull log records from the source cluster and apply them to the destination cluster corresponding to the synchronization task; it notifies the source cluster of the location information of the pulled log records applied in the destination cluster corresponding to the synchronization task and the identification information of the synchronization task.

[0075] The source cluster is used to receive notifications from synchronization tasks. The notifications include the location information of the log records that have been applied in the destination cluster corresponding to the synchronization task and the identification information of the synchronization task. Based on the identification information of the synchronization task, the minimum location information is determined from the location information of different synchronization task notifications. Log records corresponding to locations less than the determined minimum location are deleted.

[0076] The destination cluster is used to apply the log records pulled from the corresponding synchronization task and return the applied point information to the corresponding synchronization task.

[0077] In one exemplary instance, the source cluster is also used for:

[0078] Upon receiving a notification from a synchronization task, the time of the last update and deletion of points by the synchronization task is recorded; if the synchronization task does not perform any further updates within a preset time period, the identification information of the synchronization task and the point information notified by the synchronization task are automatically deleted.

[0079] Although the embodiments disclosed in this application are as described above, the content described is merely for the purpose of understanding this application and is not intended to limit this application. Any person skilled in the art to which this application pertains may make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in this application; however, the scope of patent protection of this application shall still be determined by the scope defined in the appended claims.

Claims

1. A method for achieving data synchronization, comprising: The synchronization task pulls log records from the source cluster and applies them to the destination cluster corresponding to the synchronization task; wherein, the same synchronization task corresponds to one destination cluster, different synchronization tasks correspond to different destination clusters, and the identification information of each synchronization task is unique; The synchronization task notifies the source cluster of the location information of the log records pulled and the identification information of the synchronization task in the destination cluster corresponding to the synchronization task, so that the source cluster can determine the corresponding location information based on the identification information of all synchronization tasks, find the minimum location from the location information issued by all synchronization tasks, and delete the log record corresponding to the minimum location. The implementation of the log record includes binlog. The source cluster receives a notification from the synchronization task and records the time when the synchronization task last updated and deleted the points. If the synchronization task does not update again within a preset time period, the source cluster automatically deletes the identification information of the synchronization task and the point information notified by the synchronization task.

2. A computer-readable storage medium storing computer-executable instructions for performing the data synchronization method of claim 1.

3. An apparatus for achieving data synchronization, comprising a memory and a processor, wherein, The memory stores the following instructions that can be executed by a processor: steps for performing the data synchronization method as described in claim 1.

4. A method for achieving data synchronization, comprising: The source cluster receives a notification from the synchronization task, which includes the point information of the pulled log records that have been applied in the destination cluster corresponding to the synchronization task and the identification information of the synchronization task; wherein, the same synchronization task corresponds to one destination cluster, different synchronization tasks correspond to different destination clusters, and the identification information of each synchronization task is unique. The source cluster determines the corresponding location information based on the identification information of the synchronization task, and determines the minimum location information from the location information notified by different synchronization tasks. The source cluster deletes log records corresponding to points less than the determined minimum point. The log records are implemented using binlog. The source cluster receives a notification from the synchronization task and records the time when the synchronization task last updated and deleted the point. If the synchronization task is not updated within the preset time period, delete the identification information of the synchronization task and the location information of the synchronization task notification.

5. A computer-readable storage medium storing computer-executable instructions for performing the data synchronization method of claim 4.

6. An apparatus for achieving data synchronization, comprising a memory and a processor, wherein, The memory stores the following instructions that can be executed by the processor: steps for performing the data synchronization method as described in claim 4.

7. A system for implementing data synchronization, comprising: Source cluster, multiple synchronization tasks, and multiple destination clusters, among which, A synchronization task is used to pull log records from the source cluster and apply them to the destination cluster corresponding to the synchronization task; and to notify the source cluster of the location information of the pulled log records applied in the destination cluster corresponding to the synchronization task and the identification information of the synchronization task. The same synchronization task corresponds to one destination cluster, different synchronization tasks correspond to different destination clusters, and the identification information of each synchronization task is unique. The source cluster is used to receive notifications from synchronization tasks. The notifications include the point information of the pulled log records that have been applied in the destination cluster corresponding to the synchronization task and the identification information of the synchronization task. The corresponding point information is determined based on the identification information of the synchronization task, and the minimum point information is determined from the point information of different synchronization task notifications. Log records corresponding to points smaller than the determined minimum point are deleted. The log record implementation method includes binlog. The destination cluster applies the log records pulled from the corresponding synchronization task and returns the applied point information to the corresponding synchronization task. The source cluster is also used to: receive a notification from the synchronization task, record the time when the synchronization task last updated and deleted the point; if the synchronization task does not update again within a preset time period, delete the identification information of the synchronization task and the point information notified by the synchronization task.

Images