Synchronization method, device and storage medium

Abstract

The embodiments of the present application provide a synchronization method, device and storage medium. In the synchronization method, the master node obtains the physical log and synchronizes the physical log to the standby node based on the consistency protocol during the execution of the user request. When the physical log generated by the transaction requested by the user is covered by the consistency persistence site of the cluster's physical log, the processing result of the user request can be returned. In this embodiment, there is no need to wait for the transaction to be committed before generating the logical log and synchronizing the logical log. While accurately synchronizing the physical log across nodes based on the consistency protocol, the delay generated by the log synchronization process can be reduced, thereby improving the high availability of the database and the transaction processing performance of the database.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a synchronization method, device, and storage medium. Background Art

[0002] As user data volumes increase and data security requirements evolve, users are demanding higher database availability. To improve database availability, existing database products can synchronize the logical logs corresponding to transaction operations using a master-slave replication model to back up data across different nodes. However, this logical log-based data replication method operates at the transaction level and requires synchronization after transaction commits, resulting in high latency and reduced database transaction processing performance. Therefore, a new solution is needed. Summary of the Invention

[0003] Various aspects of the present application provide a synchronization method, device, and storage medium to achieve high availability of a database and improve the transaction processing performance of the database.

[0004] An embodiment of the present application provides a synchronization method, which includes: receiving a user request; obtaining a physical log generated in the process of executing the user request; synchronizing the physical log to at least one standby node based on a consistency protocol, so that the at least one standby node stores the physical log in its respective file storage system; determining a consistent storage location of the physical log of the distributed database cluster based on a storage success message returned by the at least one standby node; and returning a processing result of the user request if the consistent storage location covers the physical log generated by the user request.

[0005] An embodiment of the present application also provides a synchronization method, which receives a physical log sent by a master node based on a consistency protocol; the physical log is generated by the master node in the process of executing a user request; the physical log is stored in a file storage system, and a storage success message is returned to the master node; in response to an instruction to upgrade to a new master node, if the storage location of the physical log in the file storage system is ahead of the current consistency storage location of the distributed database cluster, the physical log is replayed according to the current consistency storage location of the distributed database cluster.

[0006] An embodiment of the present application also provides a server, comprising: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: execute the steps in the method provided in the embodiment of the present application.

[0007] An embodiment of the present application also provides a synchronization system, which includes: a master node and at least one backup node; the master node is used to: receive user requests; obtain the physical log generated in the process of executing the user request; synchronize the physical log to at least one backup node based on a consistency protocol; determine the consistency storage location of the physical log of the distributed database cluster according to the storage success message returned by the at least one backup node; if the consistency storage location covers the physical log generated by the user request, return the processing result of the user request; any backup node is used to: receive the physical log sent by the master node based on the consistency protocol, store the physical log in the file storage system of the backup node, and return a storage success message to the master node; respond to the instruction to upgrade to a new master node, if the storage location of the physical log in the file storage system is ahead of the current consistency storage location of the distributed database cluster, replay the physical log from the file storage system according to the current consistency storage location of the distributed database cluster.

[0008] An embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the synchronization method provided in the embodiment of the present application.

[0009] In the synchronization method provided in the embodiment of the present application, the master node obtains the physical log and synchronizes the physical log to the backup node based on the consistency protocol during the execution of the user request. When the physical log generated by the transaction requested by the user is overwritten by the consistency persistence site of the cluster's physical log, the processing result of the user request can be returned. In this embodiment, there is no need to wait for the transaction to be committed before generating and synchronizing the logical log. While accurately synchronizing the physical log across nodes based on the consistency protocol, the delay generated by the log synchronization process can be reduced, thereby improving the high availability of the database and the transaction processing performance of the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0011] Figure 1 A schematic diagram of the structure of a database synchronization system provided by an exemplary embodiment of the present application;

[0012] Figure 2 A schematic diagram of a distributed database cluster from the perspective of a database kernel provided by an exemplary embodiment of the present application;

[0013] Figure 3A schematic diagram of the execution flow of a user request in a distributed database cluster provided by an exemplary embodiment of the present application;

[0014] Figure 4 A schematic diagram of cross-node synchronization of physical logs provided by an exemplary embodiment of the present application;

[0015] Figure 5 A flowchart of a process when a master node is downgraded to a standby node according to an exemplary embodiment of the present application;

[0016] Figure 6 A flowchart of a process when a standby node is upgraded to a master node according to an exemplary embodiment of the present application;

[0017] Figure 7 A flowchart of a data synchronization method provided by an exemplary embodiment of the present application executed on the master node side;

[0018] Figure 8 A flowchart of a data synchronization method provided by an exemplary embodiment of the present application, executed on a standby node side;

[0019] Figure 9 A schematic diagram of the structure of a server provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0020] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0021] The terms used in the embodiments of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The singular forms "a," "the," and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. "A plurality" generally includes at least two, but does not exclude the inclusion of at least one.

[0022] It should also be noted that the terms "include," "comprises," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a product or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such product or system. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the product or system comprising the element.

[0023] In computing, a transaction is a program execution unit that accesses and potentially updates various data items in a database. Transactions provide a mechanism for integrating all operations involved in an activity into a single, indivisible execution unit. All operations comprising a transaction are committed only if all operations execute successfully. Failure of any operation results in a rollback of the entire transaction.

[0024] Database transactions typically have characteristics such as atomicity, consistency, isolation, and durability, which are collectively referred to as ACID.

[0025] Atomicity means that all operations in a transaction will not end at any intermediate stage. If an error occurs during the execution of a transaction, it will be rolled back to the state before the transaction started.

[0026] Consistency means that the database must be in a consistent state before and after a transaction. If a transaction completes successfully, all changes in the system are correctly applied, and the system is in a valid state. If an error occurs during a transaction, all changes in the system are automatically rolled back, returning the system to its original state.

[0027] Isolation means that in a concurrent environment, when different transactions manipulate the same data simultaneously, each transaction has its own complete data space. Modifications performed by a concurrent transaction must be isolated from modifications performed by any other concurrent transaction. When a transaction views the results of a data update, the data is in the state before another transaction modified it, or in the state after another transaction modified it. Transactions cannot view data in intermediate states.

[0028] Persistence means that after a transaction successfully completes, the updates made to the database are permanently saved. Even if a system crashes, the database can be restored to the state it was in when the transaction successfully ended after the database system is restarted.

[0029] To address the technical problem that existing methods of implementing data replication based on logical logs reduce database transaction processing performance, some embodiments of this application provide a solution. This solution, based on physical logs and a consistency protocol, improves database high availability. The following, combined with the accompanying drawings, details the technical solutions provided by various embodiments of this application.

[0030] Figure 1A schematic diagram of the structure of a database synchronization system provided by an exemplary embodiment of the present application is shown in FIG. Figure 1 As shown, the database synchronization system 100 includes a database client 10 and a distributed database cluster. The database can be implemented as a distributed database such as MySQL, MongoDB or Redis, which is not limited in this embodiment.

[0031] The computing nodes in the distributed database cluster are deployed in a master-slave structure, which consists of a master node (Leader) that can perform read and write operations and at least one backup node (Follower) that can perform read operations. Figure 1 The master node 201, the backup node 202, the backup node 203 and the backup node 204 are shown. The number of backup nodes in the distributed database cluster can also be 1, 2 or 3 or more, which can be set according to the needs. Figure 1 The three spare nodes shown are only for illustrative purposes.

[0032] Any computing node can be implemented based on a server device, which can be a conventional server, a cloud server, a virtualized data center, or an elastic computing instance on the cloud, etc., and this embodiment does not impose any limitations. The server device mainly includes a processor, a hard disk, a memory, a system bus, etc., which is similar to the general computer architecture and will not be further described.

[0033] In the database synchronization system 100, computing nodes can communicate with each other using wired communication or wireless communication. The wireless communication methods include short-range communication methods such as Bluetooth, ZigBee, infrared, and WiFi (WIRE-FIdelity), as well as long-range wireless communication methods such as LORA, and wireless communication methods based on mobile networks. When connected via mobile network communication, the network standard of the mobile network can be any one of 2G (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), 5G, 5.5G, WiMax, etc.

[0034] In the embodiments of this application, Figure 1 As shown, any computing node in the computing layer is deployed with a file storage system. Therefore, the physical logs generated and received by any computing node in the computing layer do not need to be stored in the storage layer, which facilitates the synchronization of physical logs between computing nodes and the physical log playback operations of computing nodes.

[0035] Optionally, the file storage system can be a general file storage system. In addition to providing storage and playback operations for physical logs, it can also provide data storage services to other applications. For example, in some embodiments, the file storage system of the computing node can be a file storage system that complies with the POSIX (Portable Operating System Interface for Computing Systems) protocol. The POSIX protocol standardizes the service interface called by the operating system, thereby making the file storage system universal and improving the reuse performance of the file storage system.

[0036] In this embodiment, the master node 201 is primarily used to receive user requests sent by the client 10 and initiate corresponding transactions to execute the user requests. The user request may be a read request or a write request for a file in a distributed database. During the execution of the user request, the master node 201 may generate a physical log. The physical log is used to record the actual data changes of the database system on each data page in the file storage system. During the execution of the user request, the master node 201 may continuously generate and store the physical log. The storage method may be persistent storage, and the following embodiments will be described using persistent storage as an example.

[0037] During the execution of a user request, the master node 201 may obtain the physical log generated during the execution of the transaction corresponding to the user request, and may synchronize the physical log to at least one standby node in the cluster based on the consistency protocol. Figure 2 As shown, from the perspective of the database kernel, the protocol layer in the database kernel can perform consistency negotiation of the physical log based on the consistency protocol and can perform state switching on the database server layer.

[0038] In a distributed database cluster, at least one standby node can store the received physical logs, and the storage can be implemented as persistent processing of the physical logs to provide disaster recovery for the physical logs of the primary node. After any standby node receives the physical logs sent by the primary node, it can persist the physical logs in the standby node's file storage system and return a persistence success message to the primary node. When an abnormality occurs in the primary node, the standby node can quickly switch to the new primary node and quickly restore the database service based on the persisted physical logs in the local file storage system, thereby improving the high availability of the database.

[0039] Among them, the consistency protocol is used to make the various nodes of the distributed system reach a consensus on the same data value. In a distributed database cluster, the consistency protocol can be implemented as a Paxos (a consistency algorithm for message passing) protocol or a Raft (a consensus algorithm) protocol. That is, the master node 201 can synchronize the continuously generated physical logs to the at least one standby node based on the Paxos protocol or the Raft protocol. Among them, the physical logs continuously generated during the transaction processing process can be synchronized in batches or one by one, and this embodiment does not impose any restrictions.

[0040] Next, we'll use the Paxos protocol as an example to illustrate the physical log synchronization process. The Paxos protocol includes the following roles: client, proposer, acceptor, and learner. The client issues user requests to the distributed database cluster and waits for responses. The proposer proposes proposals (i.e., data values). The acceptor receives proposals and votes on them. If a proposal is approved by a majority of the cluster, it is approved. A majority in a cluster refers to more than half of the total number of decision makers in the cluster. That is, if a cluster contains N decision makers, a proposal is approved when N / 2 + 1 decision makers agree on it. The learner reads the proposal selection results from the proposer and the acceptor.

[0041] When synchronizing physical logs based on the Paxos protocol, the master node acts as a proposal initiator, proposing to synchronize the physical logs generated during the execution of transactions corresponding to user requests. Each standby node in the cluster acts as a decision maker, accepting the proposal, copying the physical logs from the master node, and persisting the copied physical logs in the local file storage system. If persistence is complete, a persistence success message is returned. If a standby node returns a persistence success message, it is considered that the standby node has approved the synchronization of the physical log. When more than half of the standby nodes in the cluster return a persistence success message for the physical log, the synchronization of the physical log is considered to have been agreed upon by a majority of the cluster. This consensus protocol ensures zero data loss during cross-node physical log synchronization, improving the disaster recovery and backup capabilities of the database cluster.

[0042] Among them, the master node can determine the consistent persistence site (i.e., consistent storage site) of the physical log of the distributed database cluster based on the persistent storage success message returned by the at least one standby node. Among them, the persistence site of the physical log is used to describe the persistence progress of the physical log. When the majority (more than half) in the cluster return the same persistence site, the master node can use the persistence site as the cluster consistent persistence site. Usually, the persistence site of the distributed database cluster may lag behind the persistence site of the master node. When the persistence site of the distributed data cluster covers all the physical logs generated by the transaction requested by the user, the master node may consider that the persistence operation of the physical log generated by the user request during the execution process has reached the consensus of the cluster majority. At this time, the transaction can be submitted and the processing result of the user request is returned to the client.

[0043] The following description will be given with reference to the accompanying drawings.

[0044] like Figure 3 As shown, in the process of executing the transaction requested by the user, the master node generates physical log i, physical log j and physical log k (i, j, k are integers). The master node can initiate consistency synchronization for the generated physical logs in turn and wait for the consistency synchronization results of the physical logs.

[0045] The persistence locations of different standby nodes may differ due to transmission speed or persistence processing progress. For example, in a cluster consisting of six compute nodes, if the persistence location of four standby nodes in the cluster is physical log i, the persistence location of another standby node is physical log j, and the persistence location of yet another standby node is physical log k, then the cluster's consistent persistence location can be determined to be physical log i.

[0046] like Figure 3 As shown, the master node can wait for the consistency synchronization process of all physical logs generated by the transaction. When the last physical log generated by the transaction (for example Figure 3 When the physical log k in the transaction is agreed upon by the majority of the cluster, the master node can commit the corresponding transaction. If the last physical log generated by the transaction is not agreed upon by the majority of the cluster, the master node can roll back the corresponding transaction.

[0047] In this embodiment, a file storage system is deployed on any computing node in the computing layer of a distributed database cluster. In the process of executing a user request, the master node obtains the physical log and synchronizes the physical log to the standby node based on the consistency protocol. When the physical log generated by the transaction requested by the user is overwritten by the consistency persistence site of the cluster's physical log, the processing result of the user request can be returned. In this embodiment, there is no need to wait for the transaction to be committed before generating and synchronizing the logical log. While accurately synchronizing the physical log across nodes based on the consistency protocol, the delay generated by the log synchronization process can be reduced, thereby improving the high availability of the database and the transaction processing performance of the database.

[0048] In some optional embodiments, to reduce the time consumption of cross-node physical log synchronization, the master node 201 can perform cross-node physical log synchronization operations in parallel while persistently storing the physical log in the local file storage system.

[0049] Optionally, the master node 201 may use an asynchronous method to persistently store the physical log generated by the transaction in the file storage system of the master node 201. Figure 3 As shown, the asynchronous mode means that after the master node 201 continuously generates the physical log, the physical log can be flushed to the cache (such as the memory cache), and the persistence process is called to write the physical log to the disk for persistent storage; after the physical log is flushed to the cache, the master node does not need to wait for the persistence to return the persistence result and can continue to execute the subsequent operations of the transaction. The persistence process can write the corresponding physical log to the disk in the order of calling. In the above process, after the physical log is flushed to the cache, it is usually necessary to wait for the persistence process to read it and write it to the disk. The master node can synchronize the physical log to at least one standby node based on the consistency protocol while the physical log is asynchronously waiting for persistent storage.

[0050] like Figure 4 As shown in the figure, after the physical log is generated, it is flushed to the cache and waits for the log SYNC (Synchronize) command. The SYNC command is used to synchronize the physical log to the file storage system of the master node to store the physical log persistently. While waiting for the log SYNC command, the master node can initiate a synchronization request for the physical log to the standby node based on the consistency protocol, that is, Figure 4This diagram illustrates Paxos synchronization. After receiving a synchronization request, the standby node retrieves the physical log to be synchronized, writes the physical log to its cache, and waits for a SYNC command to persist the log to the log file. While waiting, the primary node receives a SYNC command and persists the physical log. When a majority of the cluster reaches consensus on persisting the physical log, the primary node determines that the physical log write was successful.

[0051] Assuming that the time required for the standby node to synchronize and persist the physical log is t1, and the time required for the primary node to persist the physical log is t2, the time required for the physical log to be persisted on the primary node and synchronized across nodes is max{t1, t2}. Compared to the method of first persisting the physical log locally and then performing cross-node synchronization (requires t1 + t2), the method provided in this embodiment can shorten the cluster persistence time of the physical log.

[0052] In this implementation, the cross-node log synchronization operation can be performed in parallel with the log SYNC phase of the log in the local file storage system. There is no need to wait for the physical log to be persisted on the primary node before initiating a consistency synchronization operation to the backup node, which reduces the cluster persistence time of the log data and the impact of the cluster persistence time of the log data on the database system.

[0053] In some optional embodiments, the master node in the database cluster can be actively downgraded to a standby node when no abnormality occurs. In the following embodiments, the downgraded master node is described as computing node A. After the downgrade, computing node A can ensure the consistency of transactions on distributed nodes and the consistency between the response received by the client and the actual transaction status of the database based on the persistent physical log. Figure 5 An exemplary description is given.

[0054] Optionally, in response to the instruction to downgrade to a standby node, computing node A may determine that the persistent location has not reached cluster consensus on the physical log at the time of downgrade. Wherein, not reaching cluster consensus means that the physical log has not been synchronized and persistently stored in the majority of the cluster. Figure 5 As shown, computing node A can truncate the physical log that has not reached cluster consensus. Figure 3 For example, if physical log k has not yet reached cluster consensus when the master node is demoted to a standby node, compute node A can truncate physical log k. Typically, the persistence progress of the physical log on the master node before the demotion is greater than the cluster consensus persistence progress. After the demotion, compute node A can delete the physical log that exceeds the cluster consensus persistence progress by truncation, thereby achieving log synchronization with the new master node.

[0055] Alternatively, as Figure 5 As shown, computing node A can also determine the user requests that are in the completed execution state and waiting to be submitted when downgraded. Computing node A can detect the result of the physical log of the user request being submitted to the persistence process. The result of the submission is used to describe whether computing node A has submitted all the physical logs generated by the user request to the persistence process. If computing node A has not submitted all the physical logs generated by the user request, computing node A can return a transaction submission error prompt (i.e., handler error, HA error) to the client and clear the data in the middle of the transaction execution state. Furthermore, when the master node is downgraded, it can ensure that the response to the user request received by the client and the actual transaction state of the database are consistent.

[0056] Optionally, computing node A may also determine the user request that is in execution state during the downgrade, and cut off the user link of the user request to interrupt the transaction of the user request. Figure 5 As shown in the figure, for a user request that was in the executing state at the time of the downgrade, compute node A cannot perform the rollback operation on the transaction corresponding to the user request. The new primary node can roll back the uncommitted transaction. Therefore, compute node A clears the dirty data pages and data dictionary cache related to the transaction in the buffer pool to maintain transaction atomicity.

[0057] In some embodiments, the cluster consistency site of the physical log is smaller than the persistence site of the master node. Optionally, after being downgraded to a standby node, computing node A can also determine the physical log checkpoint after the downgrade. Typically, the physical log checkpoint lags behind the current consistency persistence site of the distributed database cluster, or the physical log checkpoint is the same as the current consistency persistence site of the distributed database cluster, thereby ensuring the security and accuracy of the data persistently stored in the database. Computing node A can replay the physical log from the file storage system of computing node A based on the checkpoint of the physical log.

[0058] For example, before computing node A is downgraded, the log persistence location of the local file storage system is physical log 6; after the downgrade, the checkpoint determined by the new master node is physical log 4. Then computing node A can replay the physical log of the local file storage system to physical log 4 to ensure the accuracy and consistency of the cluster physical log after the downgrade.

[0059] Alternatively, as Figure 6As shown, for any standby node, it can be upgraded to a master node when the master node is abnormal, or it can be actively upgraded to a master node. In response to the instruction to upgrade to a new master node, if the persistence site of the physical log in the file storage system of the new master node is ahead of the current consistency site of the cluster, the physical log can be replayed from the local file storage system according to the current consistency persistence site of the cluster, so that the persistence progress of the upgraded new master node is the same as the cluster consistency persistence progress before the upgrade. In addition, the probability of losing log data on some standby nodes with lagging persistence progress can be reduced, thereby improving the security and consistency of log data.

[0060] Optionally, the new master node can also determine the physical log at the persistence site that did not achieve cluster consistency during the upgrade from the received physical log, and truncate the physical log at the persistence site that did not achieve cluster consistency, so that the persistence progress of the new master node after the upgrade is the same as the cluster consistency persistence progress before the upgrade.

[0061] Optionally, because the physical log is not a transaction-level log, after a standby node is promoted to the new primary node, it can roll back uncommitted transactions. Rollback refers to the act of restoring a program or data to its last correct state when a program or data processing exception occurs. This transaction-based rollback operation maintains transaction atomicity during the standby node upgrade and reduces the impact of physical log replication on transaction performance. This ensures high transaction processing performance and cross-node high availability for the database while fully leveraging the physical log's low latency and excellent replay performance.

[0062] In addition to the database synchronization system described in the aforementioned embodiment, the embodiment of the present application also provides a synchronization method, which will be described below with reference to the accompanying drawings.

[0063] Figure 7 This is a flow chart of a synchronization method provided by an exemplary embodiment of the present application, which is applied to a master node in a distributed database cluster. In the distributed database cluster, any computing node in the computing layer is deployed with a file storage system. When the method is executed on the master node side of the computing layer, it may include the following steps: Figure 7 Steps shown:

[0064] Step 701: Receive a user request.

[0065] Step 702: Obtain the physical log generated during the execution of the user request.

[0066] Step 703: Based on the consistency protocol, synchronize the physical log to at least one standby node, so that the at least one standby node stores the physical log in its respective file storage system.

[0067] Step 704: Determine a consistent storage location for the physical log of the distributed database cluster according to the storage success message returned by the at least one standby node.

[0068] Step 705: If the consistent storage site covers the physical log generated by the user request, return the processing result of the user request.

[0069] Optionally, when the primary node and any standby node store physical logs, the physical logs may be persistently stored in the file storage system of the standby node.

[0070] In some exemplary embodiments, the physical log is synchronized to at least one backup node based on a consistency protocol, including: submitting the physical log to a storage process in an asynchronous manner, and waiting for the storage process to store the log in the file storage system of the primary node; in the process of asynchronously waiting for storage, the physical log is synchronized to the at least one backup node in parallel based on a consistency protocol.

[0071] In some exemplary embodiments, the method further includes: responding to an instruction to downgrade the computing node to a standby node, determining a physical log at a storage site that does not achieve cluster consistency during the downgrade; and truncating the physical log at the storage site that does not achieve cluster consistency.

[0072] In some exemplary embodiments, the method further includes: determining a first user request that has been executed and is waiting to be submitted at the time of downgrade; detecting the result of the physical log of the first user request being submitted to the storage process; if the physical log of the first user request is not completely submitted to the storage process, returning a transaction execution error prompt of the first user request, and deleting data in the middle execution state of the transaction.

[0073] In some exemplary embodiments, the method further includes: determining a physical log checkpoint after degradation; the physical log checkpoint lags behind the current consistency storage location of the distributed database cluster; and replaying the physical log from the file storage system of the master node based on the checkpoint of the physical log.

[0074] In this embodiment, a file storage system is deployed on any computing node in the computing layer of a distributed database cluster. In the process of executing a user request, the master node obtains the physical log and synchronizes the physical log to the standby node based on the consistency protocol. When the physical log generated by the transaction requested by the user is overwritten by the consistent storage site of the cluster's physical log, the processing result of the user request can be returned. In this embodiment, there is no need to wait for the transaction to be committed before generating and synchronizing the logical log. While accurately synchronizing the physical log across nodes based on the consistency protocol, the delay generated by the log synchronization process can be reduced, thereby improving the high availability of the database and the transaction processing performance of the database.

[0075] Figure 8 This is a flow chart of a synchronization method provided by an exemplary embodiment of the present application, which is applied to a standby node in a distributed database cluster. In the distributed database cluster, any computing node in the computing layer is deployed with a file storage system. When the method is executed on the standby node side of the computing layer, it may include the following steps: Figure 8 Steps shown:

[0076] Step 801: Receive a physical log sent by a master node based on a consistency protocol; the physical log is generated by the master node in the process of executing a user request.

[0077] Step 802: Store the physical log in the file storage system of the standby node, and return a storage success message to the primary node.

[0078] Step 803, in response to the instruction to upgrade to a new master node, if the storage location of the physical log in the file storage system is ahead of the current consistent storage location of the distributed database cluster, replay the physical log from the file storage system according to the current consistent storage location of the distributed database cluster.

[0079] Optionally, when the primary node and any standby node store physical logs, the physical logs may be persistently stored in the file storage system of the standby node.

[0080] In some exemplary embodiments, the method further includes: determining, from the received physical logs, the physical logs of the storage sites that did not achieve cluster consistency during the upgrade; and truncating the physical logs of the storage sites that did not achieve cluster consistency.

[0081] In some exemplary embodiments, the method further includes: determining a user request that is in an uncommitted state during the upgrade; rolling back the transaction of the user request, and deleting data in an intermediate execution state of the transaction.

[0082] In this embodiment, a file storage system is deployed on any compute node in the computing layer of a distributed database cluster. The standby node receives and stores the physical logs sent by the master node via a consistency protocol. When the standby node is promoted to the new master node, the physical logs can be replayed from the file storage system based on the cluster's consistent storage location. This ensures that the new standby node's storage location remains consistent with the cluster's consistent storage location after the master-slave switchover.

[0083] It should be noted that the execution entity of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution entity of steps 701 to 704 can be device A; for another example, the execution entity of steps 701 and 702 can be device A, and the execution entity of step 703 can be device B; and so on.

[0084] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The sequence numbers of the operations, such as 701, 702, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0085] Figure 9 FIG. 1 illustrates a schematic diagram of the structure of a server provided by an exemplary embodiment of the present application, which is applicable to the database synchronization system provided by the aforementioned embodiment. Figure 9 As shown, the server includes: a memory 901 , a processor 902 and a communication component 903 .

[0086] The memory 901 is used to store computer programs and can be configured to store various other data to support operations on the server. Examples of such data include instructions for any application program or method operating on the server.

[0087] Among them, the memory 901 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0088] The processor 902 is coupled to the memory 901 and is configured to execute a computer program in the memory 901 to execute a corresponding synchronization method.

[0089] In some exemplary embodiments, Figure 9 When the server shown is implemented as a master node in a database distributed cluster, the processor 902 is used to: receive a user request; obtain the physical log generated in the process of executing the user request; synchronize the physical log to at least one standby node based on a consistency protocol, so that the at least one standby node stores the physical log in its respective file storage system; determine the consistent storage location of the physical log of the distributed database cluster based on the storage success message returned by the at least one standby node; if the consistent storage location covers the physical log generated by the user request, return the processing result of the user request.

[0090] Optionally, when the primary node and any standby node store physical logs, the physical logs may be persistently stored in a local file storage system.

[0091] Optionally, when synchronizing the physical log to at least one backup node based on a consistency protocol, the processor 902 is specifically used to: submit the physical log to the storage process in an asynchronous manner, and wait for the storage process to store the log in the file storage system of the primary node; in the process of asynchronously waiting for storage, synchronize the physical log to the at least one backup node in parallel based on a consistency protocol.

[0092] Optionally, the processor 902 is further configured to: respond to an instruction to downgrade the computing node to a standby node, determine physical logs at storage sites that do not achieve cluster consistency during the downgrade; and truncate physical logs at storage sites that do not achieve cluster consistency.

[0093] Optionally, the processor 902 is also used to: determine the first user request that has been executed and is waiting to be submitted at the time of downgrade; detect the result of the physical log of the first user request being submitted to the storage process; if the physical log of the first user request is not completely submitted to the storage process, return a transaction execution error prompt of the first user request, and delete the data in the middle execution state of the transaction.

[0094] Optionally, the processor 902 is further used to: determine a physical log checkpoint after degradation; the physical log checkpoint lags behind the current consistent storage location of the distributed database cluster; and replay the physical log from the file storage system of the master node according to the checkpoint of the physical log.

[0095] In other exemplary embodiments, Figure 9When the server shown is implemented as a standby node in a database distributed cluster, the processor 902 is specifically used to: receive the physical log sent by the master node based on the consistency protocol; the physical log is generated by the master node in the process of executing a user request; store the physical log in the file storage system of the standby node, and return a storage success message to the master node; respond to the instruction to upgrade to a new master node, if the storage location of the physical log in the file storage system is ahead of the current consistency storage location of the distributed database cluster, replay the physical log from the file storage system according to the current consistency storage location of the distributed database cluster.

[0096] Optionally, the processor 902 is further configured to: determine, from the received physical logs, the physical logs at the storage sites that do not achieve cluster consistency during the upgrade; and truncate the physical logs at the storage sites that do not achieve cluster consistency.

[0097] Optionally, the processor 902 is further configured to: determine a user request that is in an uncommitted state during the upgrade; roll back the transaction of the user request, and delete data in an intermediate execution state of the transaction.

[0098] Further, if Figure 9 As shown, the server also includes: a communication component 903, a power supply component 904 and other components. Figure 9 Only some components are shown schematically, which does not mean that the server only includes Figure 9 Components shown.

[0099] Among them, the communication component 903 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can be implemented based on near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0100] The power supply component 904 provides power to various components of the device in which the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0101] In this embodiment, in the computing layer of a distributed database cluster, each computing node is deployed with a file storage system. During the execution of a user request, the master node obtains the physical log and synchronizes it to the backup node based on a consistency protocol. Once the physical log generated by the user's requested transaction is overwritten by the consistent persistence site of the cluster's physical log, the processing result of the user's request can be returned. In this implementation, there is no need to wait for a transaction to commit before generating and synchronizing the logical log. While accurately synchronizing the physical log across nodes based on a consistency protocol, the latency incurred during the log synchronization process can be reduced, improving the database's high availability and transaction processing performance.

[0102] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by the server in the above method embodiment.

[0103] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0105] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0107] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0108] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0109] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0110] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0111] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A synchronization method, comprising: Receive user requests; Obtaining a physical log generated during the execution of the user request; wherein the physical log is continuously generated during the execution of the user request, and the physical log is used to record actual data changes on each data page in the file storage system by the database system; Based on a consistency protocol, synchronizing the physical log to at least one standby node, so that the at least one standby node stores the physical log in a respective file storage system, wherein the storage is implemented as persistent processing of the physical log; Determining a consistent storage location for the physical log of the distributed database cluster according to the storage success message returned by the at least one standby node; the consistent storage location is used to describe the persistence progress of the physical log; If the consistent storage site overwrites the physical log generated by the user request, the processing result of the user request is returned.

2. The method according to claim 1, wherein Synchronizing the physical log to at least one standby node based on a consistency protocol includes: Submitting the physical log to a storage process in an asynchronous manner, and waiting for the storage process to store the physical log in a file storage system; During the asynchronous waiting process for storage, the physical log is synchronized to the at least one standby node in parallel based on a consistency protocol.

3. The method according to claim 1, wherein Also includes: Responding to the instruction to downgrade to a standby node, determining the physical logs that were not consistent with the cluster at the time of downgrade; The physical log for which the storage site does not achieve cluster consensus is truncated.

4. The method according to claim 3, wherein: Also includes: Determine the first user request that has been completed and is waiting to be submitted when downgrading; Detecting a result of submitting the physical log requested by the first user to the storage process; If the physical log requested by the first user is not completely submitted to the storage process, a transaction execution error prompt requested by the first user is returned, and data in the middle of the transaction execution state is deleted.

5. The method according to claim 4, wherein Also includes: Determining a physical log checkpoint after degradation; the physical log checkpoint lags behind a current consistent storage location of the distributed database cluster; The physical log is replayed from the file storage system according to the checkpoint of the physical log.

6. A synchronization method, comprising: Receive a physical log sent by the master node based on a consistency protocol; the physical log is generated by the master node in the process of executing a user request. During the execution of the user request, the physical log is continuously generated and is used to record the actual data changes of the database system on each data page in the file storage system; The physical log is stored in a file storage system, and a storage success message is returned to the master node, wherein the storage is implemented as a persistent processing of the physical log; In response to the instruction to upgrade to the new master node, if the storage location of the physical log in the file storage system is ahead of the current consistent storage location of the distributed database cluster, the physical log is replayed from the file storage system according to the current consistent storage location of the distributed database cluster; the consistent storage location is used to describe the persistence progress of the physical log.

7. The method according to claim 6, wherein: Also includes: From the received physical logs, determine the physical logs at which the storage sites do not achieve cluster consistency during the upgrade; The physical log for which cluster consistency is not achieved at the storage site is truncated.

8. The method according to claim 6, wherein: Also includes: Identify user requests that are in an unsubmitted state at the time of upgrade; The transaction requested by the user is rolled back, and the data in the middle of the transaction execution state is deleted.

9. A server, wherein: include: memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to perform the steps of the method according to any one of claims 1 to 8.

10. A synchronization system, wherein: include: A master node and at least one backup node; The master node is used to: receive user requests; Obtaining a physical log generated during the execution of the user request; wherein, during the execution of the user request, the physical log is continuously generated, and the physical log is used to record the actual data changes of the database system on each data page in the file storage system; synchronizing the physical log to at least one standby node based on a consistency protocol; determining a consistent storage location for the physical log of the distributed database cluster based on a storage success message returned by the at least one standby node, the consistent storage location being used to describe the persistence progress of the physical log; if the consistent storage location overwrites the physical log generated by the user request, returning a processing result of the user request; Any standby node is used to: receive the physical log sent by the master node based on the consistency protocol, store the physical log in the file storage system of the standby node, and return a storage success message to the master node, wherein the storage is implemented as persistent processing of the physical log; respond to the instruction to upgrade to the new master node, if the storage location of the physical log in the file storage system is ahead of the current consistency storage location of the distributed database cluster, replay the physical log from the file storage system according to the current consistency storage location of the distributed database cluster.

11. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, it can implement the synchronization method described in any one of claims 1 to 8.

Images