Data synchronization method, system, device, storage medium and computer program product

Abstract

The present disclosure provides a data synchronization method, system, device, storage medium and computer program product, relating to the technical field of computers. In the present disclosure, based on the scheme, the technical personnel can adopt the first storage space with higher read-write performance and the second storage space with lower read-write performance. In the process of log remote synchronization, if the log is missing, the missing log is first called in the second storage space and then in the first storage space, and then sent to the remote end. The second storage space with lower read-write performance has the advantage of having larger storage capacity under the same cost, which helps to improve the storage duration of the backup log. In this way, the possibility of the occurrence of the situation that the lost log cannot be read out in the storage space during the remote synchronization process is greatly reduced. Therefore, the time and resource waste caused by database reconstruction can be reduced.

Description

[0001] This disclosure claims priority to Chinese Patent Application No. 202510579411.9, filed on April 30, 2025, entitled “Method, Apparatus, Device, Storage Medium and Program Product for Data Synchronization”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of computer technology, and in particular to data synchronization methods, systems, devices, storage media, and computer program products. Background Technology

[0003] Databases often require data synchronization between master and slave business nodes. The master business node continuously transmits the logs written to disk to the slave business node.

[0004] In practical applications, ultra-long-distance synchronization is often required, with distances between the master and slave business nodes reaching thousands or even tens of thousands of kilometers. In such scenarios, the duration of a single data transmission is long, and data loss is frequently caused by network interruptions. This necessitates that the log storage nodes store logs for a sufficiently long time for retransmission. However, storage nodes used for log persistence typically have high read / write performance requirements, resulting in correspondingly high costs. To keep costs within a reasonable range, the actual capacity of storage nodes is generally limited, requiring continuous cleanup of older logs. This leads to a shorter actual log storage time for storage nodes in practical applications. Consequently, during remote synchronization, situations arise where logs are lost, even if the lost logs have already been cleaned up on the storage nodes. When this happens, database reconstruction on the slave business node is necessary, wasting significant time and resources. Summary of the Invention

[0005] This disclosure provides a data synchronization method, system, device, storage medium, and computer program product to reduce the waste of time and resources during remote database synchronization.

[0006] Firstly, a data synchronization method is provided. This method is applied to a database system, which includes a master cluster and slave clusters. The processing procedure is as follows: First, the master cluster generates logs during business processing; then, the master cluster writes the logs to a first storage space and backs them up to a second storage space; finally, the master cluster synchronizes the logs to the slave clusters so that the slave clusters can synchronize the master cluster's data based on the logs. This synchronization process may include the following steps: Step 1, the master cluster sends the logs to the slave clusters; Step 2, if the logs synchronized from the master cluster to the slave cluster are missing, the master cluster first reads the first part of the logs from the second storage space, and then reads the second part of the logs from the first storage space; Step 3, the master cluster sends the missing logs to the slave clusters.

[0007] In this disclosure, based on this scheme, technicians can employ a first storage space with higher read / write performance and a second storage space with lower read / write performance. While storing logs in the first storage space, the second storage space is used to back up the logs. During remote log synchronization, if logs are missing, the missing logs are retrieved first from the second storage space and then from the first, before being sent to the remote end. The advantage of the second storage space with lower read / write performance is that it provides a larger storage capacity at the same cost, which helps to extend the storage duration of backup logs. This significantly reduces the likelihood that lost logs cannot be read from the storage space during remote synchronization. Consequently, it reduces the waste of time and resources caused by database reconstruction.

[0008] In one possible implementation, the log is used to record changes to the data of the primary cluster. The primary cluster can back up the merge log along with the main log. The corresponding process is as follows: the merge log is backed up to a second storage space. The merge log records merged changes to the data of the primary cluster, where a merged change is the merging of multiple changes to the data of the primary cluster recorded in the log. The corresponding process for the primary cluster to read the first part of the log from the second storage space could be: the primary cluster first reads the merged log belonging to the first part of the log from the second storage space, and then reads the log belonging to the first part of the log from the second storage space.

[0009] In this way, the amount of data in the merged log is less than the sum of the data in each log before merging. Therefore, by using merged logs, the amount of data transmitted during the log retransmission process can be reduced, thereby reducing the consumption of transmission resources and improving retransmission efficiency.

[0010] In one possible implementation, after the slave cluster persists the target logs received from the master cluster, the master cluster and the slave cluster replay the target logs respectively.

[0011] This ensures that the page versions of the master and slave clusters are always synchronized, which helps prevent data corruption during master-slave switching, maintains data synchronization effectively, and prevents database rebuilds.

[0012] In one possible implementation, the process of keeping page versions synchronized can be as follows: First, the slave cluster records persistence indication information, which indicates the target log that has been persistently stored in the slave cluster's log received from the master cluster; then, the slave cluster sends the persistence indication information to the master cluster; next, the master cluster replays the target log according to the persistence indication information; finally, the slave cluster replays the target log according to the persistence indication information.

[0013] This allows for more timely updates to page versions while ensuring that the page versions of the master and slave clusters are always synchronized.

[0014] In one possible implementation, the master cluster includes service nodes and replication nodes. First, the service nodes generate the logs during business processing and write them to the first storage space. Then, the replication nodes retrieve the logs from the first storage space and send them to the slave cluster.

[0015] In this way, by having a replication node, independent of the business node, handle the log synchronization process, it is possible to prevent the synchronization process from consuming the resources of the business node and to prevent the normal business processing of the business node from being affected.

[0016] In one possible implementation, firstly, the replication node obtains synchronization point indication information from the business node, which indicates the number of logs that need to be sent for each synchronization; then, the replication node retrieves the logs from the first storage space according to the synchronization point indication information and sends the logs to the slave cluster.

[0017] This ensures that business nodes perform minimal processing and consume minimal resources during log synchronization, preventing disruption to normal business operations.

[0018] In one possible implementation, the master cluster and slave clusters are deployed in data centers in different regions.

[0019] In one possible implementation, the master cluster deletes the logs after completing log synchronization.

[0020] This allows for the timely release of storage space resources, improving resource utilization.

[0021] In one possible implementation, when a master-slave switch occurs, and the slave cluster becomes the new master cluster and the master cluster becomes the new slave cluster, the new slave cluster deletes the logs that have been persistently stored in the new master cluster.

[0022] This prevents log discrepancies between the master and slave clusters.

[0023] In a second aspect, a database system is provided, comprising at least one cluster for implementing the methods provided in the first aspect or any possible implementation thereof.

[0024] Thirdly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the method provided by the first aspect or any possible implementation thereof.

[0025] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the method provided by the first aspect or any possible implementation thereof.

[0026] Fifthly, a computer program product containing instructions is provided, which, when run by a cluster of computing devices, causes the cluster of computing devices to perform the method provided by the first aspect or any possible implementation thereof. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of a system architecture provided in an embodiment of this disclosure;

[0028] Figure 2 This is a flowchart illustrating a data synchronization method provided in an embodiment of this disclosure;

[0029] Figure 3 This is a flowchart illustrating a data synchronization method provided in an embodiment of this disclosure;

[0030] Figure 4 This is a flowchart illustrating a data synchronization method provided in an embodiment of this disclosure;

[0031] Figure 5 This is a flowchart illustrating a data synchronization method provided in an embodiment of this disclosure;

[0032] Figure 6 This is a schematic diagram of a data synchronization process provided in an embodiment of this disclosure;

[0033] Figure 7 This is a schematic diagram illustrating a process for reading lost logs provided in an embodiment of this disclosure;

[0034] Figure 8 This is a schematic diagram illustrating a process for reading lost logs provided in an embodiment of this disclosure;

[0035] Figure 9 This is a schematic diagram illustrating a process for reading lost logs provided in an embodiment of this disclosure;

[0036] Figure 10 This is a flowchart illustrating a data synchronization method provided in an embodiment of this disclosure;

[0037] Figure 11 This is a schematic diagram illustrating a process for synchronously updating page versions provided in an embodiment of this disclosure;

[0038] Figure 12 This is a schematic diagram illustrating a process for handling a situation where reading a target page from a cache pool fails, as provided in an embodiment of this disclosure.

[0039] Figure 13 This is a flowchart illustrating a data synchronization method provided in an embodiment of this disclosure;

[0040] Figure 14 This is a schematic diagram of a master-slave switching process provided in an embodiment of this disclosure;

[0041] Figure 15 This is a schematic diagram of the structure of a database system provided in an embodiment of this disclosure;

[0042] Figure 16 This is a schematic diagram of the structure of a computing device provided in an embodiment of this disclosure;

[0043] Figure 17 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this disclosure;

[0044] Figure 18 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this disclosure. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.

[0046] This disclosure provides a data synchronization method. This method is applied to a database system. For example... Figure 1 As shown, the database system may include a first cluster 110 and a second cluster 120, and may also include a first storage node 130, a second storage node 140 and a third storage node 150 (in some possible embodiments, the database system may not include these storage nodes, but instead use the storage resources configured by the first cluster 110 and the second cluster 120 themselves).

[0047] This database system can be a cloud-based database system. The nodes in the first cluster 110 and the second cluster 120, as well as the storage nodes, are all nodes established by the cloud service platform based on user needs and provided with hardware resources. These nodes can be virtual machines, physical machines, bare-metal servers, etc. The cloud service platform can be further subdivided into cloud computing platforms and cloud storage platforms. The nodes in the first cluster 110 and the second cluster 120 can be nodes established by the cloud computing platform providing hardware resources. The storage nodes can be nodes established by the cloud storage platform providing hardware resources; each storage node can belong to the same cloud storage platform or different cloud storage platforms. The aforementioned cloud computing platform and cloud storage platform can belong to the same cloud service platform or be independent platforms.

[0048] The first cluster 110 may include a first business node 111 and a first replication node 112, and may also include a first page server node 113. The first business node 111 is primarily responsible for data read / write operations and related processing. The first replication node 112 is primarily responsible for data synchronization-related processing. The first page server node 113 is primarily responsible for page-related operations, such as merging log or differential pages into a base page. Any node other than the first business node 111 may be omitted or integrated into the first business node 111.

[0049] The second cluster 120 may include a second service node 121 and a second replication node 122, and may also include a second page service node 123. The functional responsibilities of the second service node 121, the second replication node 122, and the second page service node 123 are similar to those of the first service node 111, the first replication node 112, and the first page service node 113, respectively, and will not be repeated here. Any of the above nodes, except for the second service node 121, may not be configured or may be integrated into the second service node 121.

[0050] The first storage node 130 can be a high-performance storage node bound to the first cluster 110, used for storing disk-persistent logs and pages. The third storage node 150 can be a high-performance storage node bound to the second cluster 120, used for storing disk-persistent logs and pages. The first storage node 130 and the third storage node 150 can employ high-performance distributed block storage, characterized by high read / write performance, high sequential read performance, and high random read performance, but also relatively expensive. The first storage node 130 and the third storage node 150 can be storage nodes provided by cloud storage services or independent storage devices.

[0051] The second storage node 140 can be a low-performance, low-cost storage node that can be called upon by the primary business node for long-term storage of logs, merged logs, etc. The read / write performance of the second storage node 140 is lower than that of the first storage node 130 and the third storage node 150. The second storage node 140 can use a less expensive object storage solution, such as Object Storage Service (OBS), which is characterized by higher sequential read performance, lower random read performance, and lower cost.

[0052] In a cloud-based database system, the first storage node 130 and the nodes in the first cluster 110 can be located in relatively close geographical locations, and the third storage node 150 and the nodes in the second cluster 120 can be located in relatively close geographical locations.

[0053] Assuming that in the current state, cluster 110 is the master cluster, and business node 111 is the master business node, cluster 120 is the slave cluster, and business node 121 is the slave business node. During the database system's operation, a master-slave switchover will occur when certain triggering conditions are met.

[0054] The above database system is only an example of a single slave cluster. In actual applications, the database system may also include multiple slave clusters. The architecture of other slave clusters is similar to that of the second cluster 120, and will not be described again in this embodiment.

[0055] With the continuous development of computer and network technologies, the amount of data that needs to be stored in various technical fields is increasing, and databases have also experienced tremendous growth. Especially with the support of cloud technology, database performance has been greatly optimized, offering advantages such as high performance, high availability, high reliability, high security, elastic scalability, high flexibility, and high cost-effectiveness, leading to their wider application.

[0056] Databases frequently require data synchronization between master and slave business nodes, with the master node continuously transmitting logs persisted to disk to the slave nodes. In practical applications, ultra-long-distance synchronization (such as intercontinental synchronization) is often required, with the distance between the master and slave nodes reaching thousands or even tens of thousands of kilometers. In such scenarios, the duration of a single data transmission is long, and data loss often occurs due to network interruptions. This necessitates that the log storage nodes store the logs for a sufficiently long time for retransmission.

[0057] However, storage nodes typically used for log persistence require high read / write performance, such as cloud storage services based on high-performance distributed block storage. Therefore, the corresponding costs are also relatively high. To keep costs within a reasonable range, the actual capacity of storage nodes is generally limited, requiring continuous cleanup of older logs. This results in storage nodes storing logs for a relatively short period in practice. Consequently, during remote synchronization, log loss can occur, even if the lost logs have already been cleaned up on the storage node. When this happens, database reconstruction is required on the slave business node. Database reconstruction necessitates the master business node transferring all page data to the slave business node, which wastes significant time and resources.

[0058] To address the aforementioned problems, this disclosure provides a data synchronization method applied to the database system described above. The database system may include a master cluster and slave clusters; for example, the master cluster may be the first cluster 110 described above, and the slave cluster may be the second cluster 120 described above. The master and slave clusters can be deployed in data centers in different regions, a feature also applicable to other processes in this disclosure. Figure 2 As shown, the processing flow of this method may include the following steps:

[0059] 201. The main cluster generates logs during business processing.

[0060] During the operation of the database system, the main cluster is responsible for processing user write operation commands and generating corresponding logs.

[0061] 202. The master cluster writes the logs to the first storage space and backs them up to the second storage space.

[0062] The first storage space can be the storage space configured by the business nodes in the main cluster, the storage space provided by external devices, or the storage node rented in the cloud storage system, etc. The second storage space can also have these various possibilities. Comparatively, the first storage space offers higher read / write performance than the second storage space, while the second storage space offers lower usage costs (fees) than the first storage space. In this embodiment, the solution is described in detail using a storage node rented in the cloud storage system as an example; other cases are similar and will not be elaborated further.

[0063] During the continuous generation of logs, the main cluster can flush the generated logs to disk under certain triggering conditions, writing the logs to the first storage space. If physical logs are used, the triggering conditions can be time conditions, data volume conditions, etc.; if logical logs are used, the triggering condition can be the commit of the transaction to which the log belongs. This disclosure mainly uses physical logs as an example for illustration.

[0064] In addition to flushing logs to disk, the primary cluster can also back up generated logs to a secondary storage space under a different trigger condition. The trigger conditions for flushing and backup can be the same or different.

[0065] 203. The master cluster synchronizes logs to the slave clusters so that the slave clusters can synchronize data from the master cluster based on the logs.

[0066] The synchronization process can be described as follows:

[0067] In 2031, the master cluster sends the logs to the slave cluster.

[0068] In the primary cluster, business nodes can send logs to the secondary clusters in the order they were generated, or a dedicated node responsible for log synchronization can send logs to the secondary clusters in the order they were generated. After receiving the logs, the secondary cluster can replay them in order into a page for users to read.

[0069] After the primary cluster completes log synchronization, it can delete logs. The primary cluster can first delete the logs in the first storage space, retaining the backup logs in the second storage space. A longer retention period is set for the backup logs in the second storage space; these logs are deleted only after the retention period is reached. When deleting logs in the first storage space, the logs can be deleted from the first storage space only after they have already been backed up to the second storage space.

[0070] In 2032, if the logs synchronized from the master cluster to the slave cluster are missing, the master cluster first reads the first part of the logs from the second storage space, then reads the second part of the logs from the first storage space, and then sends the missing part of the logs to the slave cluster.

[0071] When the master cluster sends logs to the slave cluster, they are sent in the order they were generated. When the master cluster detects a missing log in the slave cluster, it can determine the starting position of the missing log, which can be represented by the log sequence number (LSN). Then, the master cluster can start from that starting position and read the logs sequentially, first reading the logs from the second storage space (i.e., the first part of the logs), and then reading the logs after the first part of the logs from the first storage space (i.e., the second part of the logs), and send them to the slave cluster. The second part of the logs and subsequently generated logs can be contiguous.

[0072] Optionally, the aforementioned logs are used to record changes to the data of the primary cluster. Optionally, when backing up the logs, merge logs can also be backed up. The corresponding process is as follows: the primary cluster backs up the merge logs to the second storage space. The merge logs are used to record merged changes to the data of the primary cluster, where a merged change is the merging of multiple changes to the data of the primary cluster recorded in the logs. Accordingly, the process of reading the first part of the logs in step 2032 may include: the primary cluster first reads the merged logs belonging to the first part of the logs from the second storage space, and then reads the logs belonging to the first part of the logs from the second storage space.

[0073] The processing of storing logs in the second storage space and the processing of storing merged logs in the second storage space can be two independent processes that do not interfere with each other. The advantage of storing merged logs is that it can store a larger amount of data change information with a smaller amount of data, and can more quickly fill in missing logs when there are log gaps in the cluster. For example, if there are two logs that modify the same field in a table, the first log changes the value of the field from 4 to 5, and the second log changes the value of the field from 5 to 6, then the merged log of these two logs only needs to record the change of the field value from 4 to 6, which can reduce the amount of data by about half.

[0074] Based on the stored merged logs, when reading the first part of the logs mentioned above, the merged logs after the aforementioned start position (which can be represented by an LSN) can be read first. These merged logs correspond to logs within a certain range (which can be an LSN range), meaning that the data changes recorded in these merged logs are equivalent to the data changes recorded in the logs within that range. Then, the logs after the start position but outside that range are read from the second storage intermediate. This yields the first part of the logs, which consists of logs and merged logs.

[0075] The above-mentioned backup of logs through the second storage space and its use for recovery after the cluster logs are lost can also be performed in conjunction with other processing procedures in the embodiments of this disclosure. The processing procedures are similar and will not be described in detail here.

[0076] In this embodiment of the disclosure, based on the above scheme, technicians can employ a first storage space with higher read / write performance and a second storage space with lower read / write performance. While using the first storage space to store logs, the second storage space is used to back up the logs. During remote log synchronization, if logs are missing, the missing logs are retrieved first from the second storage space and then from the first storage space, and sent to the remote end. The advantage of the second storage space with lower read / write performance is that it has a larger storage capacity at the same cost, which helps to increase the storage duration of the backup logs. Thus, the possibility of lost logs being unreadable during remote synchronization is greatly reduced. This reduces the waste of time and resources caused by database reconstruction.

[0077] During the data synchronization process between master and slave business nodes, the business nodes often need to handle a large number of log reading and log sending operations, which will preempt the resources required for normal business processing of the business nodes.

[0078] To address this problem, this disclosure provides a data synchronization method applied to the aforementioned database system. The database system may include a master cluster and slave clusters; for example, the master cluster may be the first cluster 110 described above, and the slave cluster may be the second cluster 120 described above. The master cluster may include service nodes and replication nodes. Figure 3 As shown, the processing flow of this method may include the following steps:

[0079] 301, The business node is used to generate logs during business processing and write the logs to the first storage space.

[0080] The business node is responsible for executing the main business processes of the database, executing operation commands, generating logs, flushing logs to disk, and backing up logs.

[0081] 302. The replica node retrieves the logs from the first storage space and sends the logs to the slave cluster so that the slave cluster can synchronize the data of the master cluster based on the logs.

[0082] The replication node is responsible for performing log synchronization related processing.

[0083] Optionally, after the business node writes the logs to the first storage space, it can inform the replication node of the log storage location through the synchronization point indication information, so that the replication node can retrieve the logs. The corresponding process can be as follows: the replication node obtains the synchronization point indication information from the business node. The synchronization point indication information indicates the logs that need to be sent in one synchronization. The replication node retrieves the logs from the first storage space according to the synchronization point indication information and sends the logs to the slave cluster so that the slave cluster can synchronize the data of the master cluster according to the logs.

[0084] In the first storage space, linear storage or random access storage can be used. With linear storage, the synchronization point indicator only needs to indicate the position of the last log entry; the replication node can then retrieve the log entry to be sent based on this information. With random access storage, the synchronization point indicator can indicate the position of each log entry to be sent; the replication node can then read the log entry to be sent from each position individually.

[0085] The above Figure 2 The process involved, such as detecting missing logs in the slave cluster, reading the first part of the logs from the second storage space, reading the second part of the logs from the first storage space, and sending the missing logs to the slave cluster, can all be executed by the replication node. The specific processing is similar to step 2032 above, and you can refer to the relevant description of step 2032 above.

[0086] In this way, by having a replication node, independent of the business node, handle the log synchronization process, it is possible to prevent the synchronization process from consuming the resources of the business node and to prevent the normal business processing of the business node from being affected.

[0087] The architecture of the business nodes and replication nodes, as well as the above processing methods, can also be applied to other processing flows in the embodiments of this disclosure. The processing process is similar and will not be described again.

[0088] During data synchronization between master and slave business nodes, the logs retrieved from the slave cluster lag far behind those from the master cluster. If both the slave and master clusters immediately replay the flushed logs to the pages, the version of the pages stored in the slave cluster will always be lower than the version of the pages stored in the master cluster. This can easily lead to data corruption during master-slave failover, causing remote data synchronization to fail and requiring database rebuilding, wasting significant time and resources.

[0089] To address this problem, this disclosure provides a data synchronization method applied to the aforementioned database system. The database system may include a master cluster and slave clusters; for example, the master cluster may be the first cluster 110 described above, and the slave cluster may be the second cluster 120 described above. Figure 4 As shown, the processing flow of this method may include the following steps:

[0090] 401 indicates that the primary cluster generates logs during business processing.

[0091] This step is similar to steps 201 and 301 above, and can be found in the relevant descriptions, which will not be repeated here.

[0092] 402. The master cluster synchronizes logs to the slave clusters so that the slave clusters can synchronize data from the master clusters based on the logs.

[0093] This step is similar to steps 203 and 302 above, and can be found in the relevant descriptions, which will not be repeated here.

[0094] 403. After the slave cluster persists the target logs received from the master cluster, the master cluster and the slave cluster replay the target logs respectively.

[0095] This step can also be performed in conjunction with other processing flows in the embodiments of this disclosure. The processing flow is similar and will not be described in detail here.

[0096] Optionally, the slave cluster can inform the master cluster of the target log that it has persisted in the logs through persistence indication information. The corresponding step 403 can be processed as follows: the slave cluster records persistence indication information, which indicates the target log that the slave cluster has persisted in the logs received from the master cluster; the slave cluster sends persistence indication information to the master cluster; the master cluster replays the target logs according to persistence indication information; and the slave cluster replays the target logs according to persistence indication information.

[0097] During the continuous reception of logs from the master cluster, the slave cluster persists the logs in LSN order (i.e., flushes them to disk), while simultaneously maintaining a persistence indicator locally to record which log the slave cluster has currently persisted. The slave cluster can periodically send this persistence indicator to the master cluster. Upon receiving the persistence indicator, the master cluster can determine the latest log that the slave cluster has persisted (i.e., the target log mentioned above). The master cluster can retrieve the target log and all preceding logs that have not yet been replayed into pages, replay these logs into pages to obtain the corresponding version of the page, and store it. The slave cluster performs a similar process, retrieving the target log and all preceding logs that have not yet been replayed into pages, replaying these logs into pages to obtain the same version of the page as the master cluster, and storing it.

[0098] This enables synchronized page updates between the master and slave clusters, preventing data corruption during master-slave failover, preventing remote data synchronization failures, and ultimately preventing database rebuilds.

[0099] Optionally, when a master-slave switch occurs, and the aforementioned slave cluster becomes the new master cluster and the aforementioned master cluster becomes the new slave cluster, the new slave cluster deletes the logs that have been persistently stored in the new master cluster. This step can also be performed in conjunction with other processing flows in the embodiments of this disclosure; the processing procedures are similar and will not be described in detail here.

[0100] Master-slave failover can be triggered by a notification from the upper-level management node or by the slave cluster based on predetermined logic. During a failover, the new master cluster (the old slave cluster) obtains the current persistence indication information and sends it to the new slave cluster (the old master cluster). The persistence indication information shows that the new master cluster has persistently stored the latest log. The new slave cluster deletes all logs after this one from its locally persisted logs. If a log backup exists, it can also delete all logs after this one from the backup.

[0101] This ensures that the page versions of the master and slave clusters are always synchronized, which helps prevent data corruption during master-slave switching, maintains data synchronization effectively, and prevents database rebuilds.

[0102] The solutions provided by the above processes can be used individually or in combination. A combination of these solutions is given below. Based on the above system architecture, the processing flow of the data synchronization method provided in this disclosure embodiment is as follows: Figure 5 As shown, it includes the following steps:

[0103] 501, the first business node continuously generates logs during business processing.

[0104] During database operation, the first business node receives operation instructions from different devices. The first business node processes these instructions and generates corresponding logs. These logs include physical logs (such as redo logs) and logical logs (such as binary logs). This embodiment primarily uses physical logs for processing.

[0105] 502. Whenever the first trigger condition is met, the first business node will flush the generated logs to the first storage node.

[0106] The first triggering condition can be reaching the first cycle duration (e.g., 5 seconds), the number of logs generated reaching the first specified number (e.g., 20 logs), or the amount of log data generated reaching the first specified amount of data (e.g., 1 megabyte).

[0107] After the first business node flushes the generated logs to the first storage node, it records the LSN and storage location of the logs already stored in the first storage node. The storage location can be the offset value of the log in the first storage node. A correspondence table between LSN and storage location is established (denoted as the first correspondence table) for subsequent querying.

[0108] In the first storage node, an append log storage mode can be used. In this mode, new logs are continuously appended to the first storage node without overwriting existing logs. Logs are stored contiguously in the order of their creation; that is, generated logs are stored in consecutive physical addresses in chronological order of creation. Logs that are adjacent in creation order are physically adjacent in the first storage node. This facilitates log retrieval and simplifies the information needed to indicate the storage location of logs.

[0109] Additionally, a log deletion mechanism will be set up in conjunction with append log, as will be discussed in subsequent steps.

[0110] Because the first storage node has strong read and write performance, especially strong random read performance, the sequential storage method mentioned above can be omitted.

[0111] 503, The first business node deletes the logs in the first storage node that meet the deletion conditions.

[0112] The deletion conditions can be: the log has been backed up to the second storage node; the log has been read by the first replication node in the first storage node; the log has been stored in the first storage node for a specified period of time; the amount of log data in the first storage node has reached the upper limit, and the log is the oldest log in the first storage node; the number of logs in the first storage node has reached the upper limit, and the log is the oldest log in the first storage node, etc.

[0113] The above deletion conditions can be used individually or in any combination. The first and second deletion conditions are more conducive to reducing the storage resource occupation of the first storage node, improving resource utilization, and effectively reducing the storage resource requirements of the first storage node, thereby reducing database operating costs.

[0114] 504. Whenever the second trigger condition is met, the first business node will synchronously back up the generated logs to the second storage node, so that the logs are stored continuously in the second storage node in the order of their generation.

[0115] The second triggering condition can be reaching the second cycle duration (e.g., 5 minutes), or the number of logs generated reaches the first specified number (e.g., 500 records), or the amount of log data generated reaches the first specified amount of data (e.g., 500 MB).

[0116] Taking OBS as an example, the second storage node can be used by the first service node to request one or more OBS buckets. Within the OBS buckets, the first service node stores generated logs in consecutive physical addresses according to their generation time. Logs that are adjacent in generation order will have adjacent physical addresses in the second storage node.

[0117] The first service node can record the LSN and storage location of the logs that have been stored in the second storage node, and establish a correspondence table between LSN and storage location (denoted as the second correspondence table).

[0118] A deletion mechanism can also be set for logs stored in the second storage node. Deletion conditions can include: the log has been stored in the second storage node for a specified duration; the log data volume in the second storage node has reached its upper limit, and this log is the oldest log in the second storage node; the number of logs in the second storage node has reached its upper limit, and this log is the oldest log in the second storage node, and so on. Unlike the deletion conditions for the first storage node, because the second storage node is cheaper, the specified duration, upper limit for data volume, and upper limit for the number of logs can be longer.

[0119] 505. Whenever the third trigger condition is met, the first business node will merge the logs generated from the last time the third trigger condition was met to the current log to obtain the merged log.

[0120] The third trigger condition can be reaching the third cycle duration (e.g., 15 minutes), or the number of logs generated reaches the first specified number (e.g., 3000 records), or the amount of log data generated reaches the first specified amount of data (e.g., 3000 MB).

[0121] Taking the third cycle duration as an example, each time the third cycle duration is reached, the first business node can obtain all logs generated within this third cycle duration, merge these logs to obtain a merged log, which can also be called a delta page. The merged log (delta page) is used to represent the modifications made to the base page (or full page), and can be regarded as a log. By replaying it into the base page, the updated base page can be obtained.

[0122] 506. The first business node stores the merged logs to the second storage node, so that the merged logs are stored continuously in the second storage node in the order of their generation.

[0123] In OBS, independent storage spaces can be allocated for logs and delta pages. The first storage space stores logs, and the second storage space stores delta pages. These two storage spaces can be different OBS buckets or different regions within the same OBS bucket. In the first storage space, generated logs are stored in contiguous physical addresses in chronological order of generation. In the second storage space, generated delta pages are stored in contiguous physical addresses in chronological order of generation. Delta pages that are adjacent in generation order are physically adjacent in the second storage node.

[0124] The first service node can record the LSN range and storage location of the delta pages already stored in the second storage node, and establish a correspondence table between LSN range and storage location (denoted as the third correspondence table). The LSN range can be recorded by a start LSN and an end LSN. For example, LSN1010 to LSN2000, in this LSN range, LSN1010 is the start LSN and LSN2000 is the end LSN.

[0125] A deletion mechanism can also be set for delta pages stored in the second storage node. The deletion conditions for a delta page can be: the delta page has been stored in the second storage node for a specified duration; the amount of data in the delta page in the second storage node has reached its upper limit, and this delta page is the oldest delta page in the second storage node; the number of delta pages in the second storage node has reached its upper limit, and this delta page is the oldest delta page in the second storage node, and so on. Compared with the deletion conditions for logs in the second storage node, the specified duration for delta pages can be the same as the specified duration for logs, and the upper limit for the amount of data and the upper limit for the number of delta pages can be smaller.

[0126] Figure 6 A concrete example is given: the first business node flushes its logs to high-performance distributed block storage every 5 seconds. The numbers in the diagram represent LSN values. In high-performance distributed block storage, logs before log 504 have been deleted (cut off), and log 509 is the latest log entry. The first business node backs up its logs to OBS every 5 minutes. Logs are stored as log objects, and the storage addresses of these log objects are contiguous. Compared to high-performance distributed block storage, OBS can store a much larger number of logs. Every 15 minutes, the first business node merges the logs from those 15 minutes and backs up the merged logs (i.e., delta pages) to OBS. Delta pages are stored as delta objects, and the storage addresses of these delta objects are contiguous.

[0127] 507. Whenever the fourth triggering condition is met, the first service node generates synchronization point indication information and sends the synchronization point indication information to the first replication node.

[0128] The fourth trigger condition can be reaching the fourth cycle duration (e.g., 10 seconds), or the number of logs generated reaches the fourth specified number (e.g., 50 logs), or the amount of log data generated reaches the fourth specified amount of data (e.g., 2 MB).

[0129] Synchronization point indication information is used to indicate the storage location of the log in the first storage node. The synchronization point indication information can be a log snapshot, which includes the offset of the latest log in the first storage node.

[0130] When the fourth triggering condition is met, the first service node can retrieve the latest record's LSN and offset from the first correspondence table between the recorded LSN and the storage location, generate a log snapshot containing that LSN and offset, and send it to the first replication node. For example... Figure 3 As shown, the lsn and offset can be referred to as the final (end) lsn and end offset, respectively.

[0131] 508, The first replication node reads the log from the first storage node based on the synchronization point indication information.

[0132] After receiving a log snapshot, the first replication node can obtain its offset (denoted as offset1) and the offset from the previous log snapshot (denoted as offset0). It then reads the data from offset0 to offset1 from the first storage node. The read data represents the logs stored in the first storage node during the period from the last log snapshot to the current time. The LSNs from the previous log snapshot to the current log snapshot are the LSNs of these read logs.

[0133] After reading the logs, the first replication node can notify the first service node of the LSNs read. It can notify all LSNs read in this reading, or only the largest LSN read. If it notifies all LSNs read, the first service node can directly obtain all LSNs read in this reading. If it notifies the largest LSN read, the first service node can determine the LSNs read in this reading as the range from the largest LSN read in the previous reading to the largest LSN read in the current reading. Optionally, the first service node can delete the corresponding logs in OBS based on all LSNs read in this reading.

[0134] 509, the first replica node sends the read logs to the second replica node.

[0135] The second replication node flushes the received logs to the third storage node, such as... Figure 3As shown, the locally collected flush LSN is updated to the LSN of this log. The flush LSN is the maximum LSN of the logs flushed to disk by the second replication node. Based on the general principle of flushing logs from the cluster, it can be understood as follows: the LSNs of all logs received by the second replication node up to the flush LSN are continuous and uninterrupted, and the second replication node has not received the logs corresponding to the next LSN after the flush LSN. It can be seen that if a part of the logs is lost, even if the subsequent logs can be received, the flush LSN will not be updated (because there is an LSN gap) until the lost part of the logs is received, and then the flush LSN will continue to be updated.

[0136] 510. When the first replication node detects that the logs after the first LSN of the second replication node are lost, the second storage node sequentially reads the merged logs after the first LSN, and sequentially reads the logs after the first LSN that are outside the LSN range of the merged logs being read. The first storage node then reads the logs after the second LSN.

[0137] The second lsn is the maximum lsn of the logs read in the second storage node.

[0138] The method used by the first replica node to detect log loss in the second replica node is as follows:

[0139] The second replica node periodically sends flush lsn to the first replica node. Each time the first replica node receives a flush lsn from the second replica node, it obtains the maximum lsn of logs already sent from the first replica node to the second replica node and compares the flush lsn with this maximum lsn. If, within a specified time period, the flush lsn received by the first replica node remains unchanged and does not match the maximum lsn of logs already sent by the first replica node, the first replica node determines that logs after the first lsn (the current flush lsn) have been lost. For example, if the maximum lsn of logs already sent by the first replica node to the second replica node is 1000, and the first replica node receives flush lsn messages of 960 from the second replica node for 5 consecutive minutes (the specified time period), then the first replica node can determine that logs after lsn 960 have been lost. This specified time period can be set to be greater than or equal to the second period duration mentioned above, thus ensuring that lost logs can be read in the second storage node.

[0140] After the first replica node detects that the second replica node has lost logs after the first LSN (denoted as LSN a), it can search in the third correspondence table (the correspondence table between the LSN range and storage location of delta pages recorded in step 506) for the starting LSN (denoted as LSN b) that is closest to LSN a after LSN a. Then, it retrieves the first storage location corresponding to the LSN range of this starting LSN from the third correspondence table, and the latest stored second storage location and the ending LSN (denoted as LSN c) of the LSN range corresponding to the second storage location from the third correspondence table. Next, the first replica node sequentially reads from the first storage location to the second storage location in the second storage node. In this way, all delta pages after LSN a are read, and the LSN range corresponding to these delta pages is LSN b to LSN c.

[0141] Furthermore, if other LSNs exist between LSN a and LSN b, the first replication node obtains the next LSN of LSN a (denoted as LSN d) and the previous LSN of LSN b (denoted as LSN e). In the second mapping table (the mapping table of log LSNs and storage locations recorded in step 504), it searches for the third storage location corresponding to LSN d and the fourth storage location corresponding to LSN e. Then, in the second storage node, it sequentially reads from the third storage location up to the fourth storage location. In this way, the log between LSN a and LSN b is read. If LSN d and LSN e are equal, only one log is read. If LSN a equals LSN b, no log can be read before the already read delta page.

[0142] If LSN c is less than the largest LSN (denoted as LSN f) in the second mapping table, the first replica node obtains the next LSN (denoted as LSN g) after LSN c. In the second mapping table, it searches for the fifth storage location corresponding to LSN g and the sixth storage location corresponding to LSN f. Then, in the second storage node, it sequentially reads from the fifth storage location up to the sixth storage location. This process reads all logs after the aforementioned delta page in the second storage node. If LSN f and LSN g are equal, only one log is read. If LSN c equals LSN f, there are no logs to read after the already read delta page. LSN f is the second LSN mentioned above.

[0143] After the above processing, all logs after the first LSN in the second storage node can be read (here, the delta page can also be regarded as a log, which is a merged log obtained by merging multiple logs).

[0144] Furthermore, the first replication node can obtain the next LSN (denoted as LSN h) after LSN f, and in the first correspondence table (i.e., the correspondence table between the LSNs of the logs and their storage locations in the first storage node in step 502), it looks up the seventh storage location corresponding to LSN h, and obtains the eighth storage location (i.e., the aforementioned endoffset) in the latest received log snapshot. Then, in the first storage node, it reads the data from the seventh storage location to the eighth storage location. In this way, all the logs after the logs stored in the second storage node are obtained, and the read logs are successfully connected with the log snapshot. Subsequent processing can continue according to the processing method of steps 507-509 to continuously synchronize the logs until the next log loss occurs.

[0145] The following are some examples of the reading process described above:

[0146] like Figure 7 As shown, when the flush LSN of the second replication node is determined to be 201, in this case, firstly, in the second storage node, all delta objects after the starting LSN 201 are read, including delta object 03 and delta object 04, whose corresponding LSN ranges from 301 to 500. Then, in the second storage node, log objects between LSN 201 and LSN 301 can be read, namely log object 202, log object 202, ..., log object 300. Log objects after LSN 500 can also be read, namely log object 501, log object 502, and log object 503. Finally, in the first storage node, logs after LSN 503 can be read, namely log 504, log 505, ..., log 509.

[0147] like Figure 8As shown, when the flush LSN of the second replication node is determined to be 200, in this case, firstly, in the second storage node, all delta objects after the starting LSN of 200 are read, including delta object 02, delta object 03, and delta object 04, whose corresponding LSN ranges from 201 to 500. Then, in the second storage node, there are no more log objects between LSN 200 and LSN 201. At this point, log objects after LSN 500 can be read, namely log object 501, log object 502, and log object 503. Finally, in the first storage node, the logs after LSN 503 can be read, namely log 504, log 505, ..., log 509.

[0148] like Figure 9 As shown, when the flush LSN of the second replication node is determined to be 201, in this case, firstly, in the second storage node, all delta objects after the starting LSN 201 are read, including delta object 03 and delta object 04, whose corresponding LSN ranges from 301 to 500. Then, in the second storage node, log objects between LSN 201 and LSN 301 can be read, namely log object 202, log object 202, ..., log object 300. In the second storage node, there are no log objects after LSN 500, so no further reading is performed. Finally, in the first storage node, the logs after LSN 500 can be read, namely log 501, log 502, ..., log 506.

[0149] In the above process, the log or delta page with the largest LSN stored in the second storage node is read. In another implementation of this disclosure, the first replication node can first determine the smallest LSN of the logs stored in the first storage node, and then, in the second storage node, sequentially read the delta page between the first LSN and this smallest LSN, then sequentially read the logs between the first LSN and this smallest LSN that are outside the LSN range of the read delta page, and finally, read all the logs stored in the first storage node. The specific processing procedure is similar to the above process and will not be repeated here.

[0150] 511, the first replication node sends the read logs to the second replication node.

[0151] Step 511 is the same as step 509, and you can refer to the relevant explanation of step 509, which will not be repeated here.

[0152] In the above processing flow, optionally, the merged log can be omitted, that is, the merged log is not stored in the second storage node. Correspondingly, after determining that the log is lost, there is no need to read the merged log. Instead, the log after the first LSN is read sequentially in the second storage node, and the log after the second LSN is read in the first storage node.

[0153] In this embodiment, while a first storage node with higher read / write performance stores the flush logs, a second storage node with lower read / write performance stores the backup logs. During remote log synchronization, if log loss occurs, the lost logs are retrieved from both the first and second storage nodes and sent to the remote end. The advantage of the second storage node with lower read / write performance is that it has a larger storage capacity at the same cost, which helps to extend the storage duration of the backup logs. This significantly reduces the likelihood that lost logs cannot be retrieved from the storage nodes during remote synchronization. Consequently, it reduces the waste of time and resources caused by database reconstruction.

[0154] To address the issue of page read failure from the buffer pool, this disclosure provides a method for generating the currently needed page. The processing flow of this method is as follows: Figure 10 As shown, the process includes the following steps. The first four steps are for page persistence, and the last six steps are for handling page read failures from the buffer pool.

[0155] 1001, the second replication node continuously flushes the received logs to the third storage node.

[0156] The second replication node flushes the received logs to the third storage node and updates the locally collected flush LSNs to the LSNs of the logs. The second replication node records the LSNs and storage locations of the logs already stored on the third storage node. This storage location can be the log's offset within the third storage node, establishing a mapping table between LSNs and storage locations (denoted as the fourth mapping table). The second replication node can also synchronize the fourth mapping table to the second service node. The third storage node can use the same storage mode as the first storage node; see the relevant explanation in step 502, which will not be repeated here.

[0157] 1002, the second replica node periodically sends flush lsn to the first replica node.

[0158] Whenever the cycle time point is reached, the second replication node can obtain the current flush lsn and send it to the first replication node.

[0159] 1003. After receiving the flush lsn, the first replication node notifies both the first and second page service nodes to merge the logs up to the flush lsn into the page. This process updates the persistent page to the flush lsn version.

[0160] After receiving the flush LSN, the first replication node can confirm that the current global persistence LSN is that flush LSN. The global persistence LSN means that both the first and second clusters have persisted the logs up to that LSN.

[0161] Furthermore, the first replication node can retrieve all logs from the last flush LSN up to the current flush LSN. The first replication node can prioritize retrieving the corresponding logs from its local cache. If not found, it can check if the corresponding logs are stored on the first storage node. If not, it can read the corresponding logs from the second storage node (generally, without significant latency, the corresponding logs can be retrieved from the local cache or the first storage node; with significant latency, the corresponding logs will eventually be retrieved from the second storage node, as the second storage node typically stores logs for a long time). Then, the first replication node merges the retrieved logs to obtain a delta page and sends a first merge notification message carrying this delta page to the first page service node. The first page service node can merge the delta page carried in the first notification message into the corresponding persisted page. The persisted page can be stored on the first storage node.

[0162] Simultaneously, the first replication node can send a second merge notification message to the second replication node, which may carry the current flush LSN. The second replication node looks up the storage location corresponding to the flush LSN in the fourth mapping table, and based on that storage location, reads the logs that have not yet been merged into the page from the third storage node. Then, it merges the read logs to obtain the delta page, and sends the delta page to the second page service node. The second page service node merges the delta page into the corresponding persisted page. The persisted page can be stored in the third storage node.

[0163] 1004, the first replica node records this flush lsn as a page consolidation lsn.

[0164] The page consolidation lsn is the maximum lsn of the logs merged into the page, used to mark the version of the page persisted in the first and third storage nodes. The first replication node can maintain the page consolidation lsn. The first and second page service nodes can also maintain the page consolidation lsn.

[0165] exist Figure 11 The text provides a vivid example for steps 1001-1004 above:

[0166] The second replication node sends a `flush lsn:200` message to the first replication node. The second replication node can now confirm that both the primary and secondary clusters have completed flushing the logs for LSN 200 to disk. Further, on one hand, the second replication node merges the logs from LSNs 101 to 200 to obtain a delta page, and sends a first merge notification message carrying the delta page to the first page service node to notify it to merge the delta page into the page. On the other hand, the second replication node sends a second merge notification message carrying LSN 200 to the first replication node, so that the second replication node merges the logs from LSNs 101 to 200 to obtain a delta page, and notifies the second page service node to merge the delta page into the page. Before the merge, the page version was LSN 100; after the merge, the page version is LSN 200. In addition, the first replication node updates the pageconsolidation LSN to 200.

[0167] 1005, the first business node failed to read the target page from the buffer pool during business processing.

[0168] During business processing at the first business node, if a read operation is involved for a specific page, the latest version of that page's data needs to be retrieved and stored in the first business node's buffer pool. Pages with high access frequency ("hot") are continuously retained in the buffer pool. When a page in the buffer pool has not been accessed for an extended period, it is deleted from the buffer pool, as prolonged inactivity indicates that the page has transitioned from "hot" to "cold." When a user attempts to read that page again, the first business node will fail to read it from the buffer pool. In this case, the latest version of the page's data needs to be regenerated and written back to the buffer pool. This new page is the target page mentioned above.

[0169] 1006, the first business node sends a request to the first replication node to obtain the target page.

[0170] When the first service node fails to read the target page from the buffer pool, it obtains the page identifier of the target page, generates a retrieval request carrying the page identifier, and sends it to the first replication node.

[0171] 1007, the first replica node obtains the third and fourth LSNs.

[0172] The third LSN is the current page consolidation LSN, corresponding to the persisted version of the target page. The fourth LSN is the maximum LSN of the operation corresponding to the target page, corresponding to the latest version of the target page.

[0173] 1008, the first replica node retrieves the logs from the third LSN to the fourth LSN in the log stream sent to the second replica node, and merges the retrieved logs into the target page of the first version to obtain the target page of the second version.

[0174] The first version corresponds to the third LSN, and it is the currently persisted version. The second version corresponds to the fourth LSN, and it is the latest version.

[0175] The first replication node can obtain the first version of the target page from the first page service node, and then merge the obtained logs into the first version of the target page to obtain the second version of the target page.

[0176] 1009, the first replication node sends the target page of the second version to the first business node.

[0177] 1010, the first business node writes the target page of the second version into the buffer pool for business use.

[0178] The target page in the second version will not be persisted because the pageconsolidation lsn must match the latest flush lsn to ensure data consistency between master and slave and facilitate normal master-slave switching.

[0179] exist Figure 12 The text provides a vivid example for steps 1005-1010 above:

[0180] During business processing, the first business node misses a page A read from the buffer pool. The current version of page A is LSN 230. At this point, the first business node sends a request to the first replication node to retrieve page A. The first replication node notifies the first page service node to read the currently persisted version of page A from the first storage node, obtaining page A version LSN 200 (LSN 200 is the page consolidation LSN). The first replication node retrieves the page consolidation LSN 200 and the current version of page A, LSN 230. Then, in the log stream sent to the second replication node, the first replication node retrieves the logs from LSN 201 to LSN 230, merges the logs related to page A operations into the LSN 200 version of page A in sequence, obtaining LSN 230 version of page A, and feeds it back to the first business node. The first business node writes this LSN 230 version of page A into the buffer pool for business use.

[0181] In this way, if reading the target page from the buffer pool fails, the target page of the current version can be quickly obtained for business use.

[0182] This disclosure also provides a master-slave switching method to align logs stored on both the master and slave sides. The corresponding processing flow is as follows: Figure 13 As shown, it includes the following steps:

[0183] 1301. When the master-slave switchover event occurs, the second service node obtains the fifth LSN from the second replication node. The fifth LSN is the current flush LSN of the second replication node.

[0184] In this scenario, a master-slave switch occurs between the first and second service nodes. The master-slave switch event can be triggered by the upper-level management node sending a master-slave switch notification to the second service node (this notification is sent when the first service node is detected to have failed), or by the second service node not receiving a heartbeat message from the first service node within a specified time period, and so on.

[0185] 1302, the second service node sends a log deletion notification carrying the fifth LSN to the first service node.

[0186] 1303, The first service node deletes logs after the fifth LSN in the first storage node and the second storage node.

[0187] For delta pages in the second storage node, the first service node can determine all delta pages whose ending LSN is after the fifth LSN based on the third correspondence table, and delete these delta pages.

[0188] After the master-slave switch, the second business node (i.e., the current master business node) continues to maintain the second storage node. The logic for storing, reading, and deleting data is the same as the processing logic of the first business node before the switch.

[0189] By following the above process and deleting the logs, data duplication can be prevented after a master-slave switch.

[0190] exist Figure 14 The text provides a vivid example for steps 1301-1303 above:

[0191] When the second business node detects a master-slave switchover event, it learns that it has switched from a slave business node to a master business node (referred to as the "new master, old slave"). At this point, it can obtain the current flush lsn: 503 from the second replication node and send a log deletion notification carrying flush lsn: 503 to the first business node (referred to as the "old master, new slave"). The first business node deletes all logs after lsn 503 in the high-performance distributed storage and in OBS. For delta objects in OBS, since the largest ending lsn is less than 503, there are no delta objects that need to be deleted.

[0192] In this way, data synchronization between the master and slave ends is guaranteed during master-slave switching, preventing data duplication during the synchronization process after the switch.

[0193] This disclosure also provides a database system, such as... Figure 15 As shown, the database system includes a master cluster 1510 and a slave cluster 1520. The master cluster 1510 is used for:

[0194] Logs are generated during business processing;

[0195] Write the log to the first storage space and back up the log to the second storage space;

[0196] The logs are synchronized to the slave cluster 1520 so that the slave cluster 1520 can synchronize the data of the master cluster 1510 based on the logs;

[0197] Synchronizing the logs to the slave cluster 1520 includes:

[0198] The log is sent to the slave cluster 1520;

[0199] If the log synchronized from the master cluster 1510 to the slave cluster 1520 is missing a portion of the log, then the first portion of the log is read from the second storage space first, and then the second portion of the log is read from the first storage space.

[0200] The missing logs are sent to the slave cluster 1520.

[0201] In one possible implementation, the log is used to record changes to the data of the master cluster 1510;

[0202] The primary cluster 1510 is also used to back up the merge log to the second storage space. The merge log is used to record merge changes to the data of the primary cluster 1510. A merge change is a merge of multiple changes to the data of the primary cluster 1510 recorded in the log.

[0203] The master cluster 1510 is used to first read the merged log belonging to the first part of the log from the second storage space, and then read the log belonging to the first part of the log from the second storage space.

[0204] In one possible implementation, after the slave cluster 1520 persists the target log received from the master cluster 1510, the master cluster 1510 and the slave cluster 1520 are used to replay the target log respectively.

[0205] In one possible implementation, the slave cluster 1520 is used to record persistence indication information, which indicates that the target log that the slave cluster 1520 has persistently stored in the log of the master cluster 1510 received.

[0206] The slave cluster 1520 is used to send the persistence indication information to the master cluster 1510;

[0207] The main cluster 1510 is used to replay the target log according to the persistence instruction information;

[0208] The slave cluster 1520 is used to replay the target log according to the persistence instruction information.

[0209] In one possible implementation, the master cluster 1510 includes service nodes and replication nodes;

[0210] The business node is used to generate the log during the business processing and write the log into the first storage space;

[0211] The replication node is used to retrieve the logs from the first storage space and send the logs to the slave cluster 1520.

[0212] In one possible implementation, the replica node is used for:

[0213] Obtain synchronization point indication information from the business node, wherein the synchronization point indication information indicates the logs that need to be sent for one synchronization;

[0214] Based on the synchronization point indication information, the log is retrieved from the first storage space and sent to the slave cluster 1520.

[0215] In one possible implementation, the master cluster 1510 and the slave cluster 1520 are deployed in data centers in different regions.

[0216] In one possible implementation, the master cluster 1510 is further configured to delete the logs after the log synchronization is completed.

[0217] In one possible implementation, when a master-slave switch occurs, and the slave cluster 1520 becomes the new master cluster 1510 and the master cluster 1510 becomes the new slave cluster 1520, the new slave cluster 1520 is used to delete the logs that have been persistently stored in the new master cluster 1510.

[0218] In this embodiment of the disclosure, based on the above scheme, technicians can employ a first storage space with higher read / write performance and a second storage space with lower read / write performance. While using the first storage space to store logs, the second storage space is used to back up the logs. During remote log synchronization, if logs are missing, the missing logs are retrieved first from the second storage space and then from the first storage space, and sent to the remote end. The advantage of the second storage space with lower read / write performance is that it has a larger storage capacity at the same cost, which helps to increase the storage duration of the backup logs. Thus, the possibility of lost logs being unreadable during remote synchronization is greatly reduced. This reduces the waste of time and resources caused by database reconstruction.

[0219] Both the master cluster 1510 and the slave cluster 1520 can be implemented in software or hardware. For example, the implementation of the master cluster 1510 will be described below. Similarly, the implementation of the slave cluster 1520 can be referenced from the implementation of the master cluster 1510.

[0220] As an example of a software functional unit, a cluster, specifically the main cluster 1510, may include code running on compute instances. These compute instances can be at least one of physical hosts (computing devices), virtual machines, containers, or other computing devices. Furthermore, the aforementioned computing devices can be one or more. For example, the main cluster 1510 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed within the same region or in different regions. Similarly, the multiple hosts / virtual machines / containers used to run the code can be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0221] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a single region. Communication between two VPCs within the same region, and between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0222] As an example of a hardware functional unit, the main cluster 1510 may include at least one computing device, such as a server. Alternatively, the main cluster 1510 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0223] The main cluster 1510 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the main cluster 1510 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the main cluster 1510 can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0224] This disclosure also provides a computing device 100. For example... Figure 16 As shown, the computing device 100 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other via the bus 102. The computing device 100 may be a server or a terminal device. It should be understood that this disclosure does not limit the number of processors and memories in the computing device 100.

[0225] Bus 102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 16 The bus 102 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 102 may include a path for transmitting information between various components of the computing device 100 (e.g., memory 106, processor 104, communication interface 108).

[0226] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0227] Memory 106 may include volatile memory, such as random access memory (RAM). Memory 106 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0228] The memory 106 stores executable code, which the processor 104 executes to implement the functions of the aforementioned master cluster 1510 and slave cluster 1520, thereby realizing the data synchronization method. That is, the memory 106 stores instructions for executing the data synchronization method.

[0229] The communication interface 108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.

[0230] This disclosure also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0231] like Figure 17 As shown, the computing device cluster includes at least one computing device 100. The memory 106 of one or more computing devices 100 in the computing device cluster may store the same instructions for performing data synchronization methods.

[0232] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the data synchronization method. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for executing the data synchronization method.

[0233] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions, each used to execute a portion of the database system's functions. That is, the instructions stored in the memory 106 of different computing devices 100 can implement the functions of one or more clusters in the master cluster 1510 and slave cluster 1520.

[0234] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 18 One possible implementation is shown. For example... Figure 18 As shown, the two computing devices 100A and 100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 106 in computing device 100A stores instructions for performing the functions of the master cluster 1510. Simultaneously, the memory 106 in computing device 100B stores instructions for performing the functions of the slave cluster 1520.

[0235] Figure 18 The connection method between the computing device clusters shown can be based on the fact that in the data synchronization method provided in this disclosure, the master cluster 1510 and the slave cluster 1520 belong to different regions. Therefore, it is considered that the functions implemented by the master cluster 1510 are executed by the computing device 100A, and the functions implemented by the slave cluster 1520 are executed by the computing device 100B.

[0236] It should be understood that Figure 18 The functions of the computing device 100A shown can also be performed by multiple computing devices 100. Similarly, the functions of the computing device 100B can also be performed by multiple computing devices 100.

[0237] This disclosure also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 17 and Figure 18 The connection method of the computing device cluster. The difference is that the memory 106 of one or more computing devices 100 in the computing device cluster can store the same instructions for executing data synchronization methods.

[0238] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the data synchronization method. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for executing the data synchronization method.

[0239] This disclosure also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a data synchronization method.

[0240] This disclosure also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a data synchronization method, or instruct the computing device to perform a data synchronization method.

[0241] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data synchronization method, characterized in that, The method is applied to a database system, which includes a master cluster and a slave cluster, and the method includes: The main cluster generates logs during business processing; The main cluster writes the logs to the first storage space and backs them up to the second storage space; The master cluster synchronizes the logs to the slave clusters, so that the slave clusters can synchronize the data of the master cluster based on the logs; The process of the master cluster synchronizing the logs to the slave cluster includes: The master cluster sends the logs to the slave cluster; If the log synchronized from the master cluster to the slave cluster is missing a portion of the log, the master cluster first reads the first portion of the log from the second storage space, and then reads the second portion of the log from the first storage space. The master cluster sends the missing logs to the slave cluster.

2. The method according to claim 1, characterized in that, The log is used to record changes to the data in the main cluster; The method further includes: the primary cluster backs up the merge log to the second storage space, the merge log being used to record merge changes to the data of the primary cluster, wherein a merge change is a merge of multiple changes to the data of the primary cluster recorded in the log; The main cluster reads the first portion of logs from the second storage space, including: The main cluster first reads the merged logs belonging to the first part of the logs from the second storage space, and then reads the logs belonging to the first part of the logs from the second storage space.

3. The method according to claim 1 or 2, characterized in that, The method further includes: After the slave cluster persists the target logs received from the master cluster, the master cluster and the slave cluster respectively replay the target logs.

4. The method according to claim 3, characterized in that, After the slave cluster persists the target log received from the master cluster, the master cluster and the slave cluster respectively replay the target log, including: The slave cluster records persistence indication information, which indicates the target log that the slave cluster has persistently stored in the log of the master cluster. The slave cluster sends the persistence indication information to the master cluster; The main cluster replays the target log according to the persistence instruction information; The slave cluster replays the target log according to the persistence instruction information.

5. The method according to any one of claims 1 to 4, characterized in that, The primary cluster includes service nodes and replication nodes. The service nodes are used to generate the logs during the service processing and write the logs to the first storage space. The master cluster sends the logs to the slave cluster, including: the replication node retrieves the logs from the first storage space and sends the logs to the slave cluster.

6. The method according to claim 5, characterized in that, The replication node retrieves the logs from the first storage space and sends the logs to the slave cluster, including: The replication node obtains synchronization point indication information from the service node, and the synchronization point indication information indicates the logs that need to be sent for one synchronization. The replication node retrieves the log from the first storage space according to the synchronization point indication information and sends the log to the slave cluster.

7. The method according to any one of claims 1 to 6, characterized in that, The master cluster and the slave cluster are deployed in data centers in different regions.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: After the master cluster completes the synchronization of the logs, it deletes the logs.

9. The method according to any one of claims 3 to 8, characterized in that, The method further includes: When a master-slave switch occurs, and the slave cluster becomes the new master cluster and the master cluster becomes the new slave cluster, the new slave cluster deletes the logs that have been persistently stored in the new master cluster.

10. A database system, characterized in that, The database system includes a master cluster and a slave cluster, wherein the master cluster is used for: Logs are generated during business processing; Write the log to the first storage space and back up the log to the second storage space; The logs are synchronized to the slave cluster so that the slave cluster can synchronize the data of the master cluster based on the logs; Synchronizing the logs to the slave cluster includes: Send the logs to the slave cluster; If the log synchronized from the master cluster to the slave cluster is missing a portion of the log, then the first portion of the log is read from the second storage space first, and then the second portion of the log is read from the first storage space. The missing log entries are sent to the slave cluster.

11. The database system according to claim 10, characterized in that, The log is used to record changes to the data in the main cluster; The primary cluster is also used to back up the merge log to the second storage space. The merge log is used to record merge changes to the data of the primary cluster, wherein a merge change is a merge of multiple changes to the data of the primary cluster recorded in the log. The master cluster is used to first read the merged logs belonging to the first part of the logs from the second storage space, and then read the logs belonging to the first part of the logs from the second storage space.

12. The database system according to claim 10 or 11, characterized in that, After the slave cluster persists the target logs received from the master cluster, the master cluster and the slave cluster are used to replay the target logs respectively.

13. The database system according to claim 12, characterized in that, The slave cluster is used to record persistence indication information, which indicates the target log that the slave cluster has persistently stored in the log of the master cluster. The slave cluster is used to send the persistence indication information to the master cluster; The main cluster is used to replay the target log according to the persistence instruction information; The slave cluster is used to replay the target log according to the persistence instruction information.

14. The database system according to any one of claims 10 to 13, characterized in that, The primary cluster includes service nodes and replication nodes; The business node is used to generate the log during the business processing and write the log into the first storage space; The replication node is used to retrieve the logs from the first storage space and send the logs to the slave cluster.

15. The database system according to claim 14, characterized in that, The replication node is used for: Obtain synchronization point indication information from the business node, wherein the synchronization point indication information indicates the logs that need to be sent for one synchronization; Based on the synchronization point indication information, the log is retrieved from the first storage space and sent to the slave cluster.

16. The database system according to any one of claims 10 to 15, characterized in that, The master cluster and the slave cluster are deployed in data centers in different regions.

17. The database system according to any one of claims 10 to 16, characterized in that, The master cluster is also used to delete the logs after the log synchronization is completed.

18. The database system according to any one of claims 12 to 17, characterized in that, When a master-slave switch occurs, and the slave cluster becomes the new master cluster and the master cluster becomes the new slave cluster, the new slave cluster is used to delete the logs that have been persistently stored in the new master cluster.

19. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-9.

20. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1-9.

21. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-9.

Images