Main and standby synchronization and switching method based on nonvolatile memory

Abstract

The invention provides a main and standby synchronization and switching method based on a nonvolatile memory, and relates to the technical field of data communication, in the technical scheme provided by the invention, a check point of sql granularity is realized by recording a write and a serial number of each sql, and the write and the serial number of each sql of a main node are sent to a standby node, so that the check point of the sql granularity is realized. Seamless switching from the main node to the standby node when the main node crashes is realized by utilizing the proxy, so that a user cannot sense a transaction failure and retransmit the transaction, and a new main node does not need to execute an sql statement which is completely executed by the previous main node. Especially in a transaction which needs to frequently interact with a user or contains an sql statement with relatively high execution overhead, the method disclosed by the invention can avoid service unavailability and transaction redoing in a main / standby switching period, and greatly improves the user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of data communication, and more specifically, to a primary-backup synchronization and switching method based on non-volatile memory. Background Art

[0002] In industries such as finance and communication where high requirements are placed on data reliability and system high availability, in order to ensure service continuity and data integrity, a primary-backup architecture is usually adopted to enhance the disaster tolerance of the system. In the traditional primary-backup architecture, the primary node (Primary) is responsible for processing all transaction requests and records data changes in the WAL log. The standby node (Standby) maintains data synchronization with the primary node by reading the WAL log sent by the primary node. Under normal circumstances, the standby node does not provide services externally, and there is a certain data delay compared to the primary node. When the primary node fails due to power outage or a fault, the standby node needs to take over the service within a very short time to avoid long-term unavailability of the service. For example, in the openGauss 5.0 enterprise edition, users can implement primary-backup switching through the switchover and failover commands. Among them, the switchover command is usually used for planned switching. That is, when the primary node needs to be upgraded and maintained, the database administrator manually performs primary-backup switching through the switchover command, switching the standby node to the primary node and the primary node to the standby node. The failover command is usually used after the primary node fails, and the standby node is promoted to the primary node to ensure system availability.

[0003] Non-volatile memory (NVM), as an emerging storage technology, its representative technologies include 3D XPoint, phase change memory, magnetoresistive memory, and resistive random access memory, etc. The advantage of NVM is that it combines the byte addressing and low latency characteristics of DRAM with the non-volatility of SSD, which enables it to be used for both memory and persistent storage. In recent years, many researchers have designed and implemented NVM-based storage systems and mainly utilized its non-volatile characteristics to significantly optimize the system's fault recovery time. For example, the WAL log records which part of the database has been modified instead of the specific modified values, and allows the log to be persisted after the database modification, thereby reducing the log overhead. And after the system crashes, the WBL can restore the database to the correct state by ignoring the modifications to the database by uncommitted transactions. However, they are all limited to single-machine storage systems and do not focus on the application of NVM in the primary-backup architecture.

[0004] In the current primary and standby architecture, after the primary node fails, all transactions being executed on the primary node will be abnormally aborted. After the primary and standby switchover, the standby node needs to replay the WAL logs sent by the primary node before it can accept user connections and execute transactions as the new primary node. Since the transactions that were not completed on the primary node before were aborted prematurely, users will perceive that the transaction execution has failed and need to retransmit it to the new primary node for re-execution. Moreover, the primary and standby switchover of the vast majority of systems needs to be completed manually, and the service will be unavailable during this period, which will all lead to a poor user experience. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the purpose of the present invention is to propose a primary and standby synchronization and switching method based on non-volatile memory, including:

[0006] Step 1: The proxy receives the target transaction sent by the client, stores all the SQL statements included in the target transaction, assigns a serial number to each SQL statement, and each SQL statement calls multiple target operations; the target transaction is sent to the primary node of the primary and standby architecture of the database for database fault tolerance.

[0007] Step 2: The primary node receives the target transaction, assigns a transaction ID to the target transaction, and sends the ID of the target transaction to the proxy.

[0008] Step 3: The primary node obtains the system global globalCSN as the transaction snapshot snapshot of the target transaction, and sends the transaction snapshot snapshot and the transaction ID of the target transaction to the standby node of the primary and standby architecture of the database for database fault tolerance.

[0009] Step 4: After the standby node receives the transaction snapshot snapshot and the transaction ID of the target transaction, it starts a transaction with the same transaction snapshot as the received transaction snapshot snapshot and the same transaction ID as the target transaction.

[0010] Step 5: In the primary node, for each SQL statement, execute the target operations called by the SQL statement, record the data set modified by each target operation called by the SQL statement. After all the target operations called by the SQL statement are executed, obtain the write set write_set of the SQL statement, send the write set write_set of the SQL statement and the serial number of the SQL statement to the standby node, and at the same time send the serial number of the SQL statement and the information indicating the completion of the execution of the SQL statement to the proxy.

[0011] Step 6: After the standby node receives the write set write_set and the sequence number of the SQL statement, it persists the write set write_set to the non-volatile memory NVM. In the standby node, in the transaction that is the same as the received transaction snapshot snapshot and has the same transaction ID as the target transaction, the write set write_set is asynchronously replayed. After the asynchronous replay is completed, the sequence number of the received SQL statement is recorded;

[0012] Step 7: All SQL statements of the target transaction are executed according to Steps 5 to 6. During the execution process, if the connection status between the proxy and the primary node is normal, Step 8 is executed. During the execution process, after the primary node finishes executing the current SQL statement and sends the sequence number of the current SQL statement and the information indicating the completion of the execution of the current SQL statement to the proxy, if the connection status between the proxy and the primary node is abnormal, it indicates that the primary node has crashed, and Step 9 is executed;

[0013] Step 8: After all SQL statements of the target transaction are executed and the sequence number of the last SQL statement in the target transaction and the information indicating the completion of the execution of the last SQL statement are returned to the proxy, the system global globalCSN is obtained in the primary node as the commit sequence number csn, the commit sequence number csn is written into the transaction slot, the commit sequence number csn is synchronized with the standby node, the submission of the target transaction is completed, the global globalCSN is updated, the information indicating the completion of the processing of the target transaction is sent to the proxy, and the proxy sends the information indicating the completion of the processing of the target transaction to the client;

[0014] Step 9: The standby node replays the uncompleted write set write_set and records the SQL sequence number. The recorded SQL sequence number is used as the target sequence number. Then, the standby node is switched to the new primary node. The new primary node executes the SQL statements after the target sequence number. After the execution is completed, the sequence numbers of all SQL statements after the sequence number of the current SQL statement and the information indicating the completion of the execution of all SQL statements after the sequence number of the current SQL statement are sent to the proxy. After the sending is completed, the new primary node obtains the system global globalCSN as the commit sequence number csn, writes the commit sequence number csn into the transaction slot, completes the submission of the target transaction, updates the global globalCSN, sends the information indicating the completion of the processing of the target transaction to the proxy, and the proxy sends the information indicating the completion of the processing of the target transaction to the client.

[0015] Optionally, Step 8 specifically includes:

[0016] Step 8.1: After all the SQL statements of the target transaction are executed and the sequence number of the last SQL statement in the target transaction and the information indicating the completion of the execution of the last SQL statement are returned to the proxy, obtain the system global globalCSN as the commit sequence number csn in the primary node, write the commit sequence number csn into the transaction slot, and send the commit sequence number csn to the standby node;

[0017] Step 8.2: The standby node receives the commit sequence number csn, persists the commit sequence number csn to the non-volatile memory NVM, and determines whether all the received write_sets have been persisted. If not all the received write_sets have been persisted, wait for the unpersisted write_sets to be persisted. When all the received write_sets have been persisted, send the information indicating successful persistence to the primary node;

[0018] Step 8.3: After the primary node receives the information indicating successful persistence sent by the standby node, complete the commit of the target transaction, and send the information indicating the completion of the commit processing of the target transaction to the proxy. The proxy sends the information indicating the completion of the commit processing of the target transaction to the client.

[0019] Optionally, in step 9, the standby node replays the uncompleted write_sets of the write_set and records the SQL sequence number, uses the recorded SQL sequence number as the target sequence number, and then switches the standby node to a new primary node. The new primary node executes the SQL statements after the target sequence number. After the execution is completed, send the sequence numbers of all the SQL statements after the sequence number of the current SQL statement and the information indicating the completion of the execution of all the SQL statements after the sequence number of the current SQL statement to the proxy, specifically including:

[0020] Step A1: The proxy sends the id of the transaction being executed to the standby node;

[0021] Step A2: After the standby node receives the id of the transaction being executed, according to the transaction id, the standby node first replays the uncompleted write_sets of the write_set and records the SQL sequence number, and sends the recorded SQL sequence number as the target sequence number to the proxy;

[0022] Step A3: The proxy receives the target sequence number sent by the standby node, and sends all the SQL statements after the target sequence number in the transaction being executed and the sequence number of the current SQL statement recorded by the proxy to the standby node;

[0023] Step A4: The standby node receives all the SQL statements after the target sequence number and the sequence number of the current SQL statement recorded by the proxy. Then, the standby node is switched to a new primary node. The new primary node executes all the SQL statements after the target sequence number. After the execution is completed, the sequence numbers of all the SQL statements after the sequence number of the current SQL statement and the information indicating that the execution of all the SQL statements after the sequence number of the current SQL statement is completed are sent to the proxy.

[0024] Optionally, the target operation includes a Select operation;

[0025] Among them, the target operation called when executing the SQL statement in Step 5 specifically includes:

[0026] Search in the data table according to the row ID of the Select operation to obtain the data row corresponding to the row ID. The data row includes data and metadata. The metadata includes a version chain pointer and a timestamp. Search according to the version chain pointer to obtain the previous version corresponding to the data row, and use it as the current version. Perform a visibility judgment on the current version. Specifically, judge whether the transaction snapshot is greater than or equal to the timestamp of the current version. When the transaction snapshot is greater than or equal to the timestamp of the current version, it indicates that the current version is visible to the transaction corresponding to the Select operation, and the current version is returned as the read result. When the transaction snapshot is less than the timestamp of the current version, it indicates that the current version is not visible to the transaction corresponding to the Select operation. According to the version chain pointer included in the current version, obtain the previous version of the current version, and use the previous version of the current version as the current version. Return to execute: perform a visibility judgment on the current version.

[0027] Optionally, the target operation further includes an Update operation;

[0028] Among them, the target operation called when executing the SQL statement in Step 5 specifically includes:

[0029] Allocate a transaction slot for the target transaction in the undo log space, search for the data row corresponding to the row ID through the Update operation's row ID in the data table, lock the data row, determine whether the data row is visible, and determine whether the data row is being modified by another transaction. If the data row is not visible or the data row is being modified by another transaction, restore the data modified by all operations of the target transaction before this Update operation to the data before modification; if the data row is visible and the data row is not being modified by another transaction, copy the data row to the transaction slot in the undo log space, and then execute the Update operation to update the data row. After the update, point the timestamp in the metadata of the data row to the commit sequence number csn field of the transaction slot, and point the version chain pointer in the metadata to the data row copied by the transaction slot, and release the lock on the data row.

[0030] Optionally, the target operation further includes a Delete operation;

[0031] Among them, the target operation called by executing the sql statement in step 5 specifically includes:

[0032] Allocate a transaction slot for the target transaction in the undo log space, search for the data row corresponding to the row ID through the Delete operation's row ID in the data table, lock the data row, determine whether the data row is visible, and determine whether the data row is being modified by another transaction. If the data row is not visible or the data row is being modified by another transaction, restore the data modified by all operations of the target transaction before this Update operation to the data before modification; if the data row is visible and the data row is not being modified by another transaction, copy the data row to the transaction slot in the undo log space, and then execute the Delete operation to delete the data row. After the update, point the timestamp in the metadata of the data row to the commit sequence number csn field of the transaction slot, and point the version chain pointer in the metadata to the data row copied by the transaction slot, and release the lock on the data row.

[0033] Optionally, the target operation further includes an Insert operation;

[0034] Among them, the target operation called by executing the sql statement in step 5 specifically includes:

[0035] Allocate a transaction slot for the target transaction in the undo log space, allocate a row ID for the Insert operation in the data table, record the row ID, operation type, and row length in the undo log, lock the data row corresponding to the row ID in the data table, write the data data into the data row, point the timestamp in the metadata of the data row to the commit sequence number csn field of the transaction slot, and set the version chain pointer in the metadata to Invalid, and release the lock on the data row.

[0036] The beneficial effects produced by the above technical solutions are as follows:

[0037] Compared with the prior art, in the technical solution proposed by the present invention, a checkpoint at the SQL granularity is implemented by recording the write_set and sequence number of each SQL, and the write_set and sequence number of each SQL on the primary node are sent to the standby node. The proxy is used to achieve seamless switching from the primary node to the standby node when the primary node fails, so that users will not perceive transaction failure and retransmission, and the new primary node does not need to execute the SQL statements that have been executed by the previous primary node. Especially in transactions that need to interact frequently with users or contain SQL statements with large execution overheads, the method of the present invention will avoid service unavailability and transaction redo during the primary-standby switch, and greatly improve the user experience. Description of the Drawings

[0038] Figure 1 It is a schematic diagram of the data storage structure of Helmdb in the embodiment of the present invention;

[0039] Figure 2 It is a schematic flowchart of a primary-standby synchronization method based on non-volatile memory;

[0040] Figure 3 It is a schematic flowchart of the execution process of the Update operation in the embodiment of the present invention;

[0041] Figure 4 It is a schematic flowchart of a primary-standby switching method based on non-volatile memory in the embodiment of the present invention. Detailed Embodiments

[0042] The following combines the drawings and embodiments to further describe in detail the specific embodiments of the present invention. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0043] The purpose of the present invention is to propose a seamless primary-standby switching method that can avoid users perceiving transaction execution failure after the primary node fails and does not need to repeatedly execute the completed SQL statements. The present invention aims at the problems of poor user experience caused by users perceiving transaction execution failure during primary-standby switching and long unavailability time caused by primary-standby switching, and proposes a primary-standby synchronization and switching method based on non-volatile memory NVM. The present invention is developed on the basis of the database storage system Helmdb based on non-volatile memory, so the transaction processing method in a single machine is proposed for Helmdb.

[0044] Among them, the data storage structure of Helmdb is as Figure 1As shown. Helmdb is different from traditional disk databases that use HDD / SSD as the storage medium. Traditional disk databases store data tables in the form of pages (and read and write at the block / page granularity), while Helmdb stores the database in NVM and can perform read and write operations at the row granularity. In addition, since Helmdb stores data in the form of multiple versions, that is, the metadata of the data row tuple in Helmdb contains a timestamp and a pointer to the old version (in fact, this timestamp is also a pointer, which will be explained later), Helmdb allocates an additional space in NVM to store the old versions of the data (also known as undo logs, and a total of 2048 UndoSegments are included). This part of the old version data is organized in the form of one transaction slot after another, that is, the old versions of the data modified by the same transaction are placed together. There is a flag bit status in the metadata of the transaction slot to indicate whether this transaction has been committed, and the csn (Commit Sequence Number) of the transaction is also stored in the metadata of this transaction slot. And the timestamps of all the tuples modified by this transaction actually point to this csn. In this way, when the transaction is committed, only the 8-byte integer csn (uint64 type in the code) needs to be modified. It is this 8-byte atomic modification that ensures the atomicity of the transaction.

[0045] In addition to storing the above two parts of content in NVM, that is, the data table (the latest version) and the undo log (the old version), Helmdb also stores the index in NVM (using the PACTree index). At the same time, in order to further improve performance, Helmdb still uses DRAM to cache some attribute information of the data rows, including the flag bit and some other information, as well as the address of the data row in NVM.

[0046] The purpose of dividing the old version data (undo log) into transaction slots is that when the transaction is committed, the old version data of this entire transaction slot will completely become the old version. Even if the system crashes at this time, there is no need to roll back, which enables most of the committed transaction slots to be quickly skipped when scanning the entire undo log (old version data) for recovery.

[0047] In addition to the primary node and the standby node in the database primary-standby architecture for database fault tolerance, the present invention adds a proxy that provides services externally. That is, the database ip and port to which the client connects are actually the ip and port of the proxy. Based on this, the present invention provides a primary-standby synchronization and switching method based on non-volatile memory, combined with Figure 2 , which may include the following steps:

[0048] Step 1: The proxy receives the target transaction sent by the client, stores all the SQL statements included in the target transaction, assigns a serial number to each SQL statement, and each SQL statement invokes multiple target operations; the target transaction is sent to the primary node of the database master-slave architecture for database fault tolerance.

[0049] Step 2: The primary node receives the target transaction, assigns a transaction ID to the target transaction, and sends the ID of the target transaction to the proxy.

[0050] Step 3: The primary node obtains the system global globalCSN as the transaction snapshot snapshot of the target transaction. This snapshot is a variable of type uint64 in the code (the same type as csn), which is used to record the database state when this transaction arrives. The system global globalCSN is maintained by the system, and it ensures that transactions lower than globalCSN have been completed. The transaction snapshot snapshot and the transaction ID of the target transaction are sent to the standby node of the database master-slave architecture for database fault tolerance.

[0051] Step 4: After the standby node receives the transaction snapshot snapshot and the transaction ID of the target transaction, it starts a transaction with the same transaction snapshot as the received transaction snapshot snapshot and the same transaction ID as the target transaction.

[0052] Step 5: In the primary node, for each SQL statement, execute the target operations invoked by the SQL statement, record the set of data modified by each target operation invoked by the SQL statement. After all the target operations invoked by the SQL statement are completed, obtain the write set write_set of the SQL statement, send the write set write_set of the SQL statement and the serial number of this SQL statement to the standby node, and at the same time send the serial number of this SQL statement and the information indicating the completion of the execution of this SQL statement to the proxy.

[0053] It should be noted that the process of executing the target operation can be understood as the single-machine transaction processing of Helmdb. Next, the execution process of the target operation will be introduced.

[0054] Among them, if the target operation includes a Select operation, then the execution of the target operation invoked by the SQL statement in Step 5 specifically includes:

[0055] According to the row id of the Select operation, search in the data table to obtain the data row corresponding to this row id. The data row includes data and metadata. The metadata includes a version chain pointer and a timestamp. According to the version chain pointer, find the previous version corresponding to the data row and use it as the current version. Then, perform a visibility judgment on the current version. Specifically, judge whether the transaction snapshot snapshot is greater than or equal to the timestamp of the current version. When the transaction snapshot snapshot is greater than or equal to the timestamp of the current version, it indicates that the current version is visible to the transaction corresponding to the Select operation, and the current version is returned as the read result. When the transaction snapshot snapshot is less than the timestamp of the current version, it indicates that the current version is not visible to the transaction corresponding to the Select operation. According to the version chain pointer included in the current version, obtain the previous version of the current version, use the previous version of the current version as the current version, and return to execute: perform a visibility judgment on the current version.

[0056] Among them, when the target operation includes the Select operation, since the Select statement has no write set write_set, when sending to the standby node in step 5, the write set write_set is not sent, and only the serial number of the sql statement is sent.

[0057] Among them, the target operation also includes the Update operation. Combined with Figure 3 , the target operation called by executing the sql statement in step 5 specifically includes:

[0058] Allocate a transaction slot for the target transaction in the undo log space. Search in the data table through the row id of the Update operation (i.e., the rowid in Figure 3 ) to obtain the data row corresponding to this row id (i.e., the tuple in Figure 3 ), lock the data row, and judge whether the data row is visible. Specifically, judge whether the transaction snapshot snapshot is greater than or equal to the timestamp of the data row. When the transaction snapshot snapshot is greater than or equal to the timestamp of the data row, it indicates that the data row is visible; judge whether the data row is being modified by other transactions. The metadata of the data row contains an indication of being modified, and the timestamp of the indication is different from the transaction snapshot snapshot of the target transaction, indicating that the data row is being modified by other transactions. When the data row is not visible or the data row is being modified by other transactions, restore the data modified by all operations of the target transaction before this Update operation to the data before modification. For example, if the target transaction has 100 operations and when executing the 90th operation, the data row of the 90th operation is not visible or the data row is being modified by other transactions, then restore the data modified by all operations before the 90th operation to the data before modification.

[0059] When the data row is visible and not being modified by other transactions, copy the data row into the transaction slot in the undo log space, that is Figure 3 copy the old version to Undo in , and then perform the Update operation to update the data row. After the update, point the timestamp in the metadata of the data row to the commit sequence number csn field of the transaction slot, that is Figure 3 write the csn in , and point the version chain pointer in the metadata to the data row copied in the transaction slot, and release the lock of the data row.

[0060] Among them, the target operation also includes the Delete operation. The target operation called by executing the sql statement in step 5 specifically includes:

[0061] Allocate a transaction slot for the target transaction in the undo log space. Search in the data table through the row id of the Delete operation to obtain the data row corresponding to the row id, lock the data row, determine whether the data row is visible, and determine whether the data row is being modified by other transactions. If the data row is not visible or the data row is being modified by other transactions, restore the data modified by all operations of the target transaction before this Update operation to the data before modification; when the data row is visible and the data row is not being modified by other transactions, copy the data row into the transaction slot in the undo log space, and then perform the Delete operation to delete the data row. Among them, this deletion only sets m_isDeleted to true, rather than actually releasing the tuple space. After the update, point the timestamp in the metadata of the data row to the commit sequence number csn field of the transaction slot, and point the version chain pointer in the metadata to the data row copied in the transaction slot, and release the lock of the data row.

[0062] Among them, the target operation also includes the Insert operation. Then the target operation called by executing the sql statement in step 5 specifically includes:

[0063] Allocate a transaction slot for the target transaction in the undo log space, allocate a row id for the Insert operation in the data table, record the row id, operation type, and row length in the undo log, lock the data row corresponding to the row id in the data table, write the data data into the data row, point the timestamp in the metadata of the data row to the commit sequence number csn field of the transaction slot, and set the version chain pointer in the metadata to Invalid, and release the lock of the data row.

[0064] Among them, since the target operation is the Insert operation and this undo log has no data, only the row id, operation type, and row length can be recorded in the undo log.

[0065] Step 6: After the standby node receives the write set write_set and the sequence number of the SQL statement, it persists the write set write_set into the non-volatile memory NVM. In the standby node, in the transaction that is the same as the received transaction snapshot snapshot and has the same transaction ID as the target transaction, the write set write_set is asynchronously replayed. After the asynchronous replay ends, the sequence number of the received SQL statement is recorded;

[0066] It should be noted that in the case where the target operation includes a Select operation, since only the sequence number of the SQL statement is sent after the Select operation is completed, in Step 6, the standby node only receives the sequence number of the SQL statement. After receiving the sequence number of the SQL statement, it first determines whether the previous SQL statement has been recorded. If it has been recorded, the SQL sequence number of this Select statement is directly recorded. If the previous SQL statement has not been recorded because the replay has not been completed, it waits.

[0067] Step 7: All SQL statements of the target transaction are executed according to Steps 5 to 6. During the execution process, if the connection status between the proxy and the primary node is normal, Step 8 is executed. During the execution process, after the primary node executes the current SQL statement and sends the sequence number of the current SQL statement and the information indicating the completion of the execution of the current SQL statement to the proxy, if the connection status between the proxy and the primary node is abnormal, indicating that the primary node has crashed, Step 9 is executed;

[0068] Step 8: After all SQL statements of the target transaction are executed and the sequence number of the last SQL statement in the target transaction and the information indicating the completion of the execution of the last SQL statement are returned to the proxy, the system global globalCSN is obtained in the primary node as the commit sequence number csn. The commit sequence number csn is written into the transaction slot, the commit sequence number csn is synchronized with the standby node, the submission of the target transaction is completed, the global globalCSN is updated, the information indicating the completion of the submission of the target transaction processing is sent to the proxy, and the proxy sends the information indicating the completion of the submission of the target transaction processing to the client;

[0069] Step 8.1: After all SQL statements of the target transaction are executed and the sequence number of the last SQL statement in the target transaction and the information indicating the completion of the execution of the last SQL statement are returned to the proxy, the system global globalCSN is obtained in the primary node as the commit sequence number csn. The commit sequence number csn is written into the transaction slot, and the commit sequence number csn is sent to the standby node;

[0070] Step 8.2: The standby node receives the commit sequence number csn, persists the commit sequence number csn to the non-volatile memory NVM, and determines whether all received write sets write_set have been persisted. If not all received write sets write_set have been persisted, it waits for the unpersisted write sets write_set to be persisted. When all received write sets write_set have been persisted, it sends information indicating successful persistence to the primary node.

[0071] Step 8.3: After the primary node receives the information indicating successful persistence sent by the standby node, it completes the submission of the target transaction and sends information indicating the completion of the submission of the target transaction processing to the proxy. The proxy sends the information indicating the completion of the submission of the target transaction processing to the client.

[0072] Among them, if the asynchronous sending mode is adopted, the primary node does not need to wait for the standby node to return and can directly return to the proxy after submission.

[0073] Thus, the primary-standby synchronization scheme of the present invention realizes a checkpoint at the SQL granularity by recording the write set write_set and sequence number of each SQL, and the primary node and the standby node execute SQL statements in a transaction pipelining manner, effectively reducing the data latency of the standby node compared to the primary node. However, it should be noted that if the transaction of the primary node is rolled back due to visibility judgment, the standby node needs to be notified to roll back the modified replay as well, and return a transaction abort to the proxy to ensure the correctness of the standby node replay.

[0074] Step 9: The standby node replays the uncompleted write sets write_set and records the SQL sequence number, uses the recorded SQL sequence number as the target sequence number, and then switches the standby node to the new primary node. The new primary node executes the SQL statements after the target sequence number. After execution, it sends the sequence numbers of all SQL statements after the sequence number of the current SQL statement, and information indicating the completion of the execution of all SQL statements after the sequence number of the current SQL statement to the proxy. After sending, the new primary node obtains the system global globalCSN as the commit sequence number csn, writes the commit sequence number csn into the transaction slot, completes the submission of the target transaction, updates the global globalCSN, and sends information indicating the completion of the submission of the target transaction processing to the proxy. The proxy sends the information indicating the completion of the submission of the target transaction processing to the client.

[0075] Among them, the standby node replays the uncompleted write set write_set and records the SQL sequence number, uses the recorded SQL sequence number as the target sequence number, and then switches the standby node to the new primary node. The new primary node executes the SQL statements after the target sequence number. After the execution is completed, it sends the sequence numbers of all SQL statements after the sequence number of the current SQL statement, and the information indicating the completion of the execution of all SQL statements after the sequence number of the current SQL statement to the proxy, in combination with Figure 4 , specifically including:

[0076] Step A1: The proxy sends the id of the transaction being executed to the standby node.

[0077] Step A2: After receiving the id of the transaction being executed, the standby node replays the uncompleted write set write_set according to the transaction id, records the SQL sequence number, and sends the recorded SQL sequence number as the target sequence number to the proxy.

[0078] Step A3: The proxy receives the target sequence number sent by the standby node, and sends all SQL statements after the target sequence number in the transaction being executed, and the sequence number of the current SQL statement recorded by the proxy to the standby node.

[0079] Step A4: The standby node receives all SQL statements after the target sequence number and the sequence number of the current SQL statement recorded by the proxy, then switches the standby node to the new primary node. The new primary node executes all SQL statements after the target sequence number. After the execution is completed, it sends the sequence numbers of all SQL statements after the sequence number of the current SQL statement, and the information indicating the completion of the execution of the sequence numbers of all SQL statements after the sequence number of the current SQL statement to the proxy.

[0080] For example, in combination with Figure 4 , if transaction 1 has 9 SQL statements, when the primary node fails after the 6th SQL statement is executed and the result is returned to the proxy, at this time the sequence number of the current SQL statement is 6, and the sequence number recorded by the proxy is 6. After the proxy senses it, it sends the transaction id (1) to the standby node. After receiving it, the standby node first replays the uncompleted write set write_set of transaction 1 and records the SQL sequence number. If the recorded sequence number is 5 at this time, the target sequence number is 5, and this sequence number (5) is returned to the proxy. The proxy sends the SQL statements with sequence numbers 6-9 and the current SQL statement sequence number (6) recorded by the proxy to the standby node, and then switches the standby node to the new primary node. The new primary node executes the SQL statements with sequence numbers 6-9. After the execution is completed, it sends the sequence numbers 7-9 and the information indicating the completion of the execution of the SQL statements 7-9 to the proxy.

[0081] The advantage of the present invention implemented on the NVM-based database storage system is that since the single-machine transaction of Helmdb is an in-place update in NVM, there is no need to persist the write_set of each sql anymore. However, in the traditional DRAM+SSD disk storage system, the write_set of each sql needs to be written to the SSD in the form of a log to ensure data is not lost after the primary node fails and shuts down. Frequent writes to the SSD will undoubtedly significantly reduce the performance of the primary node. Therefore, the present invention has more advantages in the NVM-based storage system.

[0082] Compared with the prior art, in the technical solution proposed by the present invention, a checkpoint at the sql granularity is achieved by recording the write_set and sequence number of each sql, and seamless switching from the primary node to the standby node is realized by using a proxy, so that users will not perceive transaction failure and retransmission, and the new primary node does not need to execute the sql statements that have been executed by the previous primary node. Especially in transactions that need to interact frequently with users or contain sql statements with large execution overheads, the method of the present invention will avoid service unavailability and transaction redo during the primary-standby switch, and greatly improve the user experience.

[0083] The above description is only the preferred embodiments of the present disclosure and the description of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by mutually replacing the above features with the (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for primary and standby synchronization and switching based on non-volatile memory, characterized in that, Including: Step 1: The proxy receives the target transaction sent by the client, stores all the SQL statements included in the target transaction, assigns a serial number to each SQL statement, and each SQL statement calls multiple target operations; Send the target transaction to the primary node of the database master-slave architecture for database fault tolerance; Step 2: The primary node receives the target transaction, assigns a transaction ID to the target transaction, and sends the ID of the target transaction to the proxy; Step 3: The primary node obtains the system global CSN as the transaction snapshot of the target transaction, and sends the transaction snapshot and the transaction ID of the target transaction to the standby node of the database master-slave architecture for database fault tolerance; Step 4: After the standby node receives the transaction snapshot and the transaction ID of the target transaction, start a transaction whose transaction snapshot is the same as the received transaction snapshot and whose transaction ID is the same as the transaction ID of the target transaction; Step 5: In the primary node, for each SQL statement, execute the target operations called by the SQL statement, record the set of data modified by each target operation called by the SQL statement. After all the target operations called by the SQL statement are executed, obtain the write set of the SQL statement. Send the write set of the SQL statement and the serial number of the SQL statement to the standby node, and at the same time send the serial number of the SQL statement and the information indicating the completion of the execution of the SQL statement to the proxy; Step 6: After the standby node receives the write set and the serial number of the SQL statement, persist the write set to the non-volatile memory NVM. In the transaction in the standby node that is the same as the received transaction snapshot and the same as the transaction ID of the target transaction, asynchronously replay the write set. After the asynchronous replay ends, record the received serial number of the SQL statement; Step 7: All the SQL statements of the target transaction are executed according to Steps 5 to 6. If the connection status between the proxy and the primary node is normal during the execution, execute Step 8. During the execution, after the primary node executes the current SQL statement and sends the serial number of the current SQL statement and the information indicating the completion of the execution of the current SQL statement to the proxy, if the connection status between the proxy and the primary node is abnormal, indicating that the primary node has crashed, execute Step 9; Step 8: After all the SQL statements of the target transaction are executed and the sequence number of the last SQL statement in the target transaction and the information indicating the completion of the execution of the last SQL statement are returned to the proxy, obtain the system global globalCSN as the commit sequence number csn in the primary node, write the commit sequence number csn into the transaction slot, synchronize the commit sequence number csn with the standby node, complete the commit of the target transaction, update the global globalCSN, send the information indicating the completion of the handling and commit of the target transaction to the proxy, and the proxy sends the information indicating the completion of the handling and commit of the target transaction to the client; Step 9: The standby node replays the incomplete write_set of the write set and records the SQL sequence number, uses the recorded SQL sequence number as the target sequence number, then switches the standby node to the new primary node. The new primary node executes the SQL statements after the target sequence number. After the execution is completed, it sends the sequence numbers of all SQL statements after the sequence number of the current SQL statement and the information indicating the completion of the execution of all SQL statements after the sequence number of the current SQL statement to the proxy. After the sending is completed, the new primary node obtains the system global globalCSN as the commit sequence number csn, writes the commit sequence number csn into the transaction slot, completes the commit of the target transaction, updates the global globalCSN, sends the information indicating the completion of the handling and commit of the target transaction to the proxy, and the proxy sends the information indicating the completion of the handling and commit of the target transaction to the client.

2. The primary and standby synchronization and switching method based on non-volatile memory according to claim 1, wherein Step 8 specifically includes: Step 8.1: After all the SQL statements of the target transaction are executed and the sequence number of the last SQL statement in the target transaction and the information indicating the completion of the execution of the last SQL statement are returned to the proxy, obtain the system global globalCSN as the commit sequence number csn in the primary node, write the commit sequence number csn into the transaction slot, and send the commit sequence number csn to the standby node; Step 8.2: The standby node receives the commit sequence number csn, persists the commit sequence number csn to the non-volatile memory NVM, and determines whether all the received write_sets of the write set have been persisted. If not all the received write_sets of the write set have been persisted, wait for the unpersisted write_sets of the write set to be persisted. In the case where all the received write_sets of the write set have been persisted, send the information indicating the success of persistence to the primary node; Step 8.3: After the primary node receives the information indicating the success of persistence sent by the standby node, complete the commit of the target transaction, and send the information indicating the completion of the handling and commit of the target transaction to the proxy. The proxy sends the information indicating the completion of the handling and commit of the target transaction to the client.

3. A master-slave synchronization and switching method based on non-volatile memory according to claim 1, characterized in that In step 9, the standby node replays the write set write_set that has not been completed for replay, records the SQL sequence number, uses the recorded SQL sequence number as the target sequence number, and then switches the standby node to the new master node. The new master node executes the SQL statements after the target sequence number. After the execution is completed, the sequence numbers of all SQL statements after the sequence number of the current SQL statement, and the information indicating that all SQL statements after the sequence number of the current SQL statement have been executed are sent to the proxy proxy. Specifically, it includes: Step A1: The proxy proxy sends the id of the transaction being executed to the standby node; Step A2: After receiving the id of the transaction being executed, the standby node replays the write set write_set that has not been completed for replay according to the transaction id, records the SQL sequence number, and sends the recorded SQL sequence number as the target sequence number to the proxy proxy; Step A3: The proxy proxy receives the target sequence number sent by the standby node, and sends all SQL statements after the target sequence number in the transaction being executed, and the sequence number of the current SQL statement recorded by the proxy proxy to the standby node; Step A4: The standby node receives all SQL statements after the target sequence number, and the sequence number of the current SQL statement recorded by the proxy proxy, and then switches the standby node to the new master node. The new master node executes all SQL statements after the target sequence number. After the execution is completed, the sequence numbers of all SQL statements after the sequence number of the current SQL statement, and the information indicating that the sequence numbers of all SQL statements after the sequence number of the current SQL statement have been executed are sent to the proxy proxy.

4. A method for primary and standby synchronization and switching based on non-volatile memory according to claim 1, characterized in that The target operation includes a Select operation; Among them, the target operation called when executing the SQL statement in step 5 specifically includes: According to the row id of the Select operation, search in the data table to obtain the data row corresponding to the row id. The data row includes data and metadata. The metadata includes a version chain pointer and a timestamp. According to the version chain pointer, find the previous version corresponding to the data row, and use it as the current version. Perform a visibility judgment on the current version. Specifically, judge whether the transaction snapshot snapshot is greater than or equal to the timestamp of the current version. When the transaction snapshot snapshot is greater than or equal to the timestamp of the current version, it indicates that the current version is visible to the transaction corresponding to the Select operation, and the current version is returned as the read result. When the transaction snapshot snapshot is less than the timestamp of the current version, it indicates that the current version is not visible to the transaction corresponding to the Select operation. According to the version chain pointer included in the current version, obtain the previous version of the current version, and use the previous version of the current version as the current version. Return and execute: Perform a visibility judgment on the current version.

5. The primary-backup synchronization and switching method based on non-volatile memory according to claim 1, wherein The target operation also includes an Update operation; Among them, the target operation called when executing the SQL statement in step 5 specifically includes: Allocate a transaction slot for the target transaction in the undo log space. Search in the data table using the row ID of the Update operation to obtain the data row corresponding to this row ID. Lock the data row. Determine whether the data row is visible and whether it is being modified by another transaction. In the case where the data row is not visible or it is being modified by another transaction, restore the data modified by all operations of the target transaction before this Update operation to the data before modification; in the case where the data row is visible and it is not being modified by another transaction, copy the data row to the transaction slot in the undo log space, then execute the Update operation to update the data row. After the update, set the timestamp in the metadata of the data row to point to the commit sequence number csn field of the transaction slot, and set the version chain pointer in the metadata to point to the data row copied by the transaction slot. Release the lock on the data row.

6. A method for primary and standby synchronization and switching based on non-volatile memory according to claim 1, characterized in that The target operation further includes the Delete operation; Among them, the target operation called by executing the sql statement in step 5 specifically includes: Allocate a transaction slot for the target transaction in the undo log space. Search in the data table using the row ID of the Delete operation to obtain the data row corresponding to this row ID. Lock the data row. Determine whether the data row is visible and whether it is being modified by another transaction. In the case where the data row is not visible or it is being modified by another transaction, restore the data modified by all operations of the target transaction before this Update operation to the data before modification; in the case where the data row is visible and it is not being modified by another transaction, copy the data row to the transaction slot in the undo log space, then execute the Delete operation to delete the data row. After the update, set the timestamp in the metadata of the data row to point to the commit sequence number csn field of the transaction slot, and set the version chain pointer in the metadata to point to the data row copied by the transaction slot. Release the lock on the data row.

7. A master-slave synchronization and switching method based on non-volatile memory according to claim 1, characterized in that The target operation further includes the Insert operation; Among them, the target operation called by executing the sql statement in step 5 specifically includes: Allocate a transaction slot for the target transaction in the undo log space. Allocate a row ID for the Insert operation in the data table. Record the row ID, operation type, and row length in the undo log. Lock the data row corresponding to the row ID in the data table. Write the data data into the data row. Set the timestamp in the metadata of the data row to point to the commit sequence number csn field of the transaction slot, and set the version chain pointer in the metadata to Invalid. Release the lock on the data row.

Images