Database migration method and system

Abstract

Embodiments of the present specification provide a database migration method and system, wherein the database migration method includes: a source database node suspending an executing transaction, performing a snapshot operation, and obtaining a database snapshot page; after completing the snapshot operation, the source database node uses the database snapshot page to resume executing the transaction based on a copy-on-write mode; routing the transaction received after the snapshot operation to a destination database node; the destination database node uses the database snapshot page to execute the received transaction based on a copy-on-write mode; the source database node sends modified content to the destination database node, causing the destination database node to perform logical replay using the modified content, where the modified content corresponds to the write operation performed by the source database node after the snapshot operation; and upon determining that the logical replay of the destination database node has been successfully completed, the source database node commits the transaction corresponding to the modified content.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a database migration method. Background Art

[0002] In a shared storage-based database, database data is distributed across a shared storage area. One or more database instances run on compute nodes, providing read and write services. These instances are called RW nodes. Meanwhile, multiple read-only database instances, called RO nodes, share storage with the RW nodes and provide consistent read-only query services.

[0003] To improve elasticity, public clouds need to scale up and down. For example, during peak service periods, RW nodes with high computing demands can be migrated to newly added or idle physical machines. However, current database migration solutions cause service interruptions and incur high resource overhead. Therefore, finding a database migration solution that avoids service interruptions and reduces resource overhead has become a pressing issue. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a database migration method. One or more embodiments of this specification also relate to a database migration system, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art.

[0005] According to a first aspect of an embodiment of the present specification, a database migration method is provided, comprising: suspending an executing transaction at a source database node, performing a snapshot operation, and obtaining a database snapshot page; after completing the snapshot operation, the source database node using the database snapshot page to resume executing the transaction based on a copy-on-write mode; routing transactions received after the snapshot operation to a destination database node, the destination database node and the source database node being database instances storing database data based on a shared storage area; executing the received transactions based on the copy-on-write mode using the database snapshot page; the source database node sending modified content to the destination database node, causing the destination database node to perform logical replay using the modified content, the modified content being modified content corresponding to a write operation performed by the source database node after the snapshot operation; and upon determining that the logical replay of the destination database node has been successfully completed, the source database node committing the transaction corresponding to the modified content.

[0006] According to a second aspect of an embodiment of the present specification, a database migration method is provided, which is applied to a source database node and includes: suspending an executing transaction, performing a snapshot operation, and obtaining a database snapshot page; after completing the snapshot operation, resuming transaction execution using the database snapshot page in a copy-on-write mode, wherein the snapshot operation is used to route transactions received after the snapshot operation to a destination database node; wherein the database snapshot page is further used to cause the destination database node to execute the received transactions using the database snapshot page in a copy-on-write mode after receiving the transactions, wherein the destination database node and the source database node are database instances storing database data based on a shared storage area; sending modified content of the write operation performed after the snapshot operation to the destination database node, causing the destination database node to perform logical replay using the modified content; and upon determining that the logical replay of the destination database node has successfully completed, committing the transaction corresponding to the modified content.

[0007] According to a third aspect of the embodiments of this specification, a database migration method is provided, which is applied to a destination database node and includes: receiving a transaction, the transaction being a transaction routed to the destination database node after a source database node performs a snapshot operation and obtains a database snapshot page, the destination database node and the source database node being database instances storing database data based on a shared storage area; executing the received transaction using the database snapshot page in a copy-on-write mode, the database snapshot page being further configured to enable the source database node to resume executing the transaction using the database snapshot page in the copy-on-write mode; obtaining modified content corresponding to a write operation performed by the source database node after the snapshot operation; and performing logical replay using the modified content, so that the source database node commits the transaction corresponding to the modified content upon determining that the logical replay of the destination database node has successfully completed.

[0008] According to a fourth aspect of the embodiments of this specification, a database migration system is provided, comprising: a source database node implemented by the database migration method as described in any embodiment of this specification, and a destination database node implemented by the database migration method as described in any embodiment of this specification.

[0009] According to a fifth aspect of an embodiment of this specification, a computing device is provided, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implements the source database node and / or destination database node in the database migration system described in any embodiment of this specification.

[0010] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the steps of the database migration method described in any embodiment of this specification are implemented.

[0011] According to a seventh aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned database migration method.

[0012] One embodiment of the present specification implements a database migration method. When a database service needs to be migrated from a source database node to a destination database node, the source database node suspends the currently executing transaction, performs a snapshot operation, and obtains a database snapshot page. After the snapshot operation is completed, not only can the source database node resume executing transactions based on a copy-on-write mode, but transactions received after the snapshot operation can also be routed to the destination database node, causing the destination database node to execute transactions based on a copy-on-write mode. In this way, the source database node and the destination database node each execute transactions before and after the snapshot without interrupting database service. Furthermore, the source database node sends the modified content of the write operation performed after the snapshot operation to the destination database node, causing the destination database node to perform logical replay using the modified content. This not only ensures that the destination database node obtains globally consistent database data, but also reduces the amount of data transferred and resource overhead associated with the migration. Furthermore, upon successful completion of the logical replay on the destination database node, the source database node commits the transactions running on it, completing the migration of the database service. This implements a database migration method that does not interrupt database service during the migration process and reduces resource overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a schematic diagram of a cloud-native database application scenario of a database migration method provided in one embodiment of this specification;

[0014] Figure 2 is a flowchart of a database migration method provided by one embodiment of this specification;

[0015] Figure 3 This is a schematic diagram of a write operation in a copy-on-write mode provided by an embodiment of this specification;

[0016] Figure 4 This is a log processing flow chart during the database migration process provided by an embodiment of this specification;

[0017] Figure 5 This is an online migration architecture based on shared storage space provided by an embodiment of this specification;

[0018] Figure 6This is a signaling interaction diagram of a database migration method provided by an embodiment of this specification;

[0019] Figure 7 This is a flowchart of a database migration method applied to a source database node provided by one embodiment of this specification;

[0020] Figure 8 This is a structural diagram of a database migration device configured on a source database node provided by one embodiment of this specification;

[0021] Figure 9 This is a flowchart of a database migration method applied to a destination database node provided by one embodiment of this specification;

[0022] Figure 10 This is a structural diagram of a database migration device configured on a destination database node provided by one embodiment of this specification;

[0023] Figure 11 This is a schematic diagram of the structure of a database migration system provided by one embodiment of this specification;

[0024] Figure 12 This is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0025] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0026] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0027] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0028] First, the terms involved in one or more embodiments of this specification are explained.

[0029] Tuple: A tuple represents a row of records in a database table.

[0030] Database pages, the database is stored in pages, storing table Tup le and index items.

[0031] The heap is a memory area distinct from the stack, global data area, and code area. It allows programs to dynamically request memory space as needed during runtime.

[0032] Database files. The database creates files through the file system to store the heap and index data of the table, and organizes and manages them by page.

[0033] Buffer pool is used by the database to read database pages from the hard disk into the memory cache pool for accelerated access. When memory resources are tight, the cache pool uses a certain elimination algorithm to replace pages.

[0034] A checkpoint is an event that, when activated, triggers a flush operation. This flush operation involves the database writer process writing dirty data from the cache pool to data files. When a checkpoint event is initiated periodically or by a user, the database instance writes all dirty data in the cache pool back to persistent storage and creates a corresponding checkpoint.

[0035] MVCC (Multi-version Concurrency Control) is commonly used in relational databases to implement isolation in transaction ACI D. ACI D stands for atomicity, consistency, isolation, and durability.

[0036] Snapshot isolation: Databases typically provide transaction snapshot isolation based on MVCC, meaning that a transaction can only see modifications made by other transactions committed before it started. To support snapshot isolation, each transaction is assigned a start timestamp at the start of the transaction and a commit timestamp when the transaction commits. A transaction T1 can see modifications made by another committed transaction T2 if and only if T1's start timestamp is later than T2's commit timestamp.

[0037] Database Log (WAL): The database uses a write-ahead log mechanism to ensure persistence and fault recovery. This mechanism means that changes are not written directly to the database file, but rather to the database log. If a transaction fails, the record in the database log is ignored, undoing the change. If the transaction succeeds, it is written back to the database file at a later time, committing the change.

[0038] Database node: refers to a database instance that runs database code and provides database services.

[0039] Compute node: A physical or virtual machine where a database instance runs. A compute node can run one or more database instances.

[0040] In the cloud computing field, one approach to improving elasticity is to scale capacity through virtual machine migration. For example, during peak service hours, virtual machines located on RW nodes with high computing demand can be migrated to newly added or idle physical machines. This virtual machine migration method is transparent to RW nodes and can improve their computing power, but it can cause a certain amount of interruption in database services. Furthermore, virtual machine migration incurs significant resource overhead. Another approach is to maintain the transaction connection and suspend routing the transaction's statement execution objects to the source RW node when a migration command is issued. This approach waits for the execution of the statement execution objects currently running on the source RW node to complete. The source RW node then flushes all dirty data. Finally, the proxy server connects to the newly started destination RW node, reloads the transaction execution parameter environment, and routes the new statement execution objects to the destination RW node, thus achieving database migration. However, this method still has significant downtime, specifically: (1) downtime waiting for the execution of the statement execution object being executed to end. For statement execution objects with long statements (such as complex queries), the downtime is very long; (2) before the destination RW node takes over, it is necessary to wait for the source RW node to complete all dirty page flushing operations. The downtime is proportional to the size of the dirty page set and the IO execution time of the dirty page flushing, resulting in a long downtime.

[0041] In view of this, this specification provides a database migration method, and this specification also involves a database migration device, a database migration system, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0042] See also Figure 1 , Figure 1 A schematic diagram of a cloud native database application scenario of a database migration method provided by an embodiment of this specification is shown. Figure 1 As shown in the figure, a cloud-native database with a storage and computing separation architecture based on cloud facilities includes a storage layer and a computing layer. It supports elastic scaling of multi-tenants, such as adding and reducing computing nodes, online migration of database services, and database software upgrades and operations. The storage layer can be distributed shared storage, supporting elastic storage scalability, and is used to store database data. The computing layer can include one or more computing nodes, and one computing node can run one or more database instances. Among them, at least one database instance is Figure 1 The source database node shown, at least one other database instance is Figure 1 The destination database node is shown. The source database node provides read and write services. The database data of the source and destination database nodes is distributed on shared storage. The destination database node is a database instance started based on the same database directory as the source database node. In addition, multiple read-only database nodes can be created. Read-only database nodes provide consistent read-only query services.

[0043] based on Figure 1 In the cloud-native database scenario shown, when a database service needs to be migrated from a source database node to a destination database node, the source database node suspends ongoing transactions, performs a snapshot operation, and obtains a database snapshot page. A proxy server routes transactions received after the snapshot operation to the destination database node. After the snapshot operation is completed, the source database node resumes transaction execution, utilizing the database snapshot page in a copy-on-write mode. The destination database node executes the received transactions, utilizing the database snapshot page in a copy-on-write mode. Consequently, after the snapshot operation completes, the source and destination database nodes form a dual execution mode. To ensure that the destination database node obtains globally consistent database data, the source database node sends the modified content of the write operation performed after the snapshot operation to the destination database node, allowing the destination database node to perform logical replay using the modified content. The source database node also performs a dirty flush operation after the snapshot operation, flushing dirty data in the source database node's cache to the shared storage area. The source database node commits the transaction corresponding to the modified content when determining that the destination database node has successfully completed the logical playback of the modified content.

[0044] It should be noted that the database migration method provided in the embodiments of this specification is not limited to the above application scenarios and can be implemented based on various types of databases, such as databases with shared storage, supporting MVCC and logical replication.

[0045] It can be seen from the above application scenarios that the source database node creates a database snapshot page on the shared storage area. Based on the write-time copy mode, the source database node and the destination database node can each execute transactions before and after the snapshot on both sides during the migration process without interrupting the database service, thereby eliminating the impact of service interruption and downtime during the migration process. Moreover, the source database node sends the modified content of the write operation performed after the snapshot operation to the destination database node, so that the destination database node uses the modified content for logical replay. The source database node performs a dirty flush operation to flush the dirty data of the database snapshot page and the dirty data of the first write-time copy file to the shared storage area, thereby ensuring that the destination database node obtains globally consistent database data. Then, when the logical replay of the destination database node is successfully completed, the source database node commits the transaction running on it, thereby completing the migration of the database service, thereby realizing a database migration method that does not require interruption of the database service during the migration process.

[0046] See also Figure 2 , Figure 2 A flowchart of a database migration method provided according to an embodiment of this specification is shown, which specifically includes the following steps.

[0047] Step 202: The source database node suspends the currently executing transaction, performs a snapshot operation, and obtains a database snapshot page.

[0048] The suspension refers to the operating system suspending the execution of the transaction process.

[0049] The snapshot operation refers to an operation of obtaining a read-only view of a database at a certain point in time.

[0050] The database snapshot page is a read-only view of the database at a certain point in time.

[0051] A transaction is a sequence of database operations used to access and potentially manipulate various data items in a database. The operations of a transaction are either all executed or none executed.

[0052] The source database is snapshotted to obtain a consistent snapshot of the source database, also known as a database snapshot page. This snapshot contains all database pages at the snapshot point, including database pages in shared storage and dirty pages in the local cache pool that have not yet been written to shared storage. Database pages include heap table file and index file pages. The destination database node can read each database snapshot page from the source database node over a high-speed network.

[0053] For example, if a database service needs to be migrated from a source database node to a destination database node, a consistent snapshot can be taken at a specific time. If the source database node is currently executing transactions A and B when the snapshot is ready, transactions A and B are paused before taking the snapshot.

[0054] In order to further avoid database service interruption, the destination database node can be created before the source database node performs the snapshot operation, so that the routing transactions received after the snapshot operation can be routed to the destination database node as soon as possible without waiting. After the destination database node is successfully created, the source database node performs a consistency snapshot operation. It should be noted that in some application scenarios, only one read-write node is allowed in the database system. In this case, before the proxy server switches the transaction to the destination database node, the destination database node can only be started as a read-only database node. After the proxy server switches the transaction to the destination database node, there will be no new transaction requests to the source database node. At this time, the destination database node is set to a read-write node with read-write functions.

[0055] For example, when the proxy server determines that the database service needs to be migrated from the source database node to the destination database node, a destination database node is created for the source database node on the destination machine based on shared storage. The process of creating a destination database node based on shared storage may include: starting a database instance as the destination database node based on the same database directory as the source database node on the destination machine. After the destination database node is started, it may temporarily not accept any SQL processing and be in an idle state. Before executing the snapshot operation, the source database node first suspends the execution of all database transactions, which can also be understood as suspending the process or thread used to execute the transaction, and then persists the database log records cached in the memory to the shared storage area, performs the snapshot operation and marks that the entire database has been snapshotted, and then resumes the execution of the transactions of the source database node.

[0056] To ensure that the database state is consistent at the time of the snapshot, that is, that correct reading and writing can be performed based on the database snapshot, the source database node's transaction is suspended until the atomic modification operation currently in execution is completed. For example, the modification operation on a B-tree index is an atomic modification operation. Specifically, in one or more embodiments of this specification, the method may further include, before the source database node suspends the currently executing transaction:

[0057] The source database node determines whether there is an atomic modification operation being executed;

[0058] If there is an atomic modification operation being executed, wait for the atomic modification operation to be completed, and then enter the step of suspending the transaction being executed by the source database node;

[0059] If there is no atomic modification operation being executed, the process proceeds to a step in which the source database node suspends the transaction being executed.

[0060] For example, the following three operations can be defined as atomic modification operations:

[0061] (1) A complete table insert / delete / update operation (including corresponding index modification);

[0062] (2) The transaction commit / rollback operation being executed;

[0063] (3) Elimination / flushing of the local cache Buffer Pool.

[0064] For another example, in one or more embodiments of the present specification, logic for checking snapshot requests can be inserted into the database code to suspend transactions, such as a function check_snapshot_request (check_snapshot_request is an exemplary function name for a function that checks snapshot requests; in actual applications, the function names may vary). In this embodiment, the step of inserting the function can be performed by the database kernel of the source database node. Specifically, the database kernel determines whether there is an atomic modification operation being executed. If so, the function is inserted at the position after the atomic modification operation ends. When the transaction executes the check_snapshot_request function, the transaction is suspended. When all transactions of the source database node are suspended, a snapshot request is started and a snapshot point is set.

[0065] In addition, to avoid waiting for a large amount of dirty IO, snapshot operations can be avoided when the source database node is performing a checkpoint.

[0066] Step 204: After completing the snapshot operation, the source database node uses the database snapshot page to resume executing transactions based on the copy-on-write mode.

[0067] The Copy-on-Write (COW) mode is an optimization strategy in the field of computer programming. The core idea is: if multiple callers request the same resource at the same time, they will jointly obtain the same pointer to point to the same resource. Only when a caller tries to modify the content of the resource will a dedicated copy be actually copied to the caller, while the original resource seen by other callers remains unchanged. This process is transparent to other callers. The advantage of the Copy-on-Write mode is that if the caller does not modify the resource, no copy will be created, so multiple callers can share the same resource only when reading. According to the method provided in the embodiment of this specification, the source database node and the destination database node can perform read operations on the same database snapshot page. During a write operation, the database snapshot page related to the write operation will be copied to a copy-on-write file (also called a COW file), and the write operation will be performed in the copy-on-write file.

[0068] For example, after completing the snapshot operation, the source database node performs a write operation in a first copy-on-write file by copying the relevant database snapshot page based on the copy-on-write mode.

[0069] After the snapshot operation completes, newly arriving transactions are routed to the destination database node, while existing transactions already routed to the source database node continue to run on the source database node. Newly arriving transactions refer to transactions that arrive after the snapshot operation begins, including those that arrive during and after the snapshot operation. A transaction can consist of multiple SQL statements. After an old transaction is paused and resumed, its subsequent SQL statements continue to be routed to the source database node for execution.

[0070] Step 206: Routing the transaction received after the snapshot operation to a destination database node, where the destination database node and the source database node are database instances storing database data based on a shared storage area.

[0071] In one or more embodiments of the present specification, the routing policy of the proxy server can be used to make the commit timestamp of the transaction routed to the destination database node greater than the start timestamp of the old transaction routed to the source database node, so that the modifications of the new transaction of the destination database node are not visible to the old transaction of the source database node under snapshot isolation.

[0072] After the snapshot operation is completed, the source database node and the destination database node simultaneously execute the reading and writing of their respective transactions.

[0073] Step 208: The destination database node executes the received transaction based on the copy-on-write mode using the database snapshot page.

[0074] For example, the destination database node performs a write operation in the second copy-on-write file by copying the relevant database snapshot page based on the copy-on-write mode.

[0075] After the snapshot operation is completed, the source and destination database nodes each execute transactions received before and after the snapshot is taken. Therefore, after the snapshot operation is completed, the source and destination database nodes are actually reading and writing simultaneously. To avoid conflicts caused by simultaneous reading and writing, the source and destination database nodes use the copy-on-write mode for write operations.

[0076] Next, the read operation and the write operation of the transaction executed simultaneously by the source database node and the destination database node are respectively described in detail.

[0077] For the read operation of the source database node, the source database node first determines whether the data to be read is in the local cache pool. If so, the data to be read is obtained from the local cache pool. If not, it is determined whether the data to be read is in the first write-time copy file of the source database node. If so, the data to be read is obtained from the first write-time copy file and placed in the local cache pool. Otherwise, the data to be read is obtained from the database snapshot page stored in the shared storage area and placed in the local cache pool.

[0078] For the read operation of the destination database node, since the content of the latest database snapshot page is on the source database node, it may not have been flushed to the shared storage. Therefore, in this case, the destination database node can first determine whether the data to be read is in the local cache pool. If so, obtain the data to be read from the local cache pool. If not, determine whether the data to be read is in the second write-time copy file of the destination database node. If so, obtain the data to be read from the second write-time copy file and put it into the local cache pool. Otherwise, obtain the data to be read from the source database node and put it into the local cache pool.

[0079] When a destination database node needs to obtain a database snapshot page from a source database node, it can request the page from the source database node over a high-speed network. If the page is in the source database node's cache pool, the page data is directly returned. If the page is no longer in the cache pool, the data can be retrieved using one of the following methods: the source database node reads the page from shared storage and returns it to the destination database node, or the destination database node reads the page from shared storage itself.

[0080] To improve read and write efficiency, you can also create an index for the copy-on-write file, also known as a COW index. When searching for data in the copy-on-write file, you can first search the corresponding index. Specifically, in conjunction with the above-mentioned read operation process, when the source database node reads a database page that is not in the local cache pool, it first searches the corresponding COW index. If the COW index indicates that the database page to be read exists, the COW page is read from the COW file and stored in the local cache pool. Otherwise, the database snapshot page is read and stored in the local cache pool.

[0081] For a write operation of a source database node, the source database node uses the database snapshot page to resume executing the transaction based on a copy-on-write mode, including:

[0082] If the write operation of the transaction resumed by the source database node is the first write operation on the first database file after the snapshot operation, create a corresponding first copy-on-write file for the first database file, where the first database file is the database file of the source database node;

[0083] If the write operation of the transaction resumed by the source database node is the first write operation on a first database snapshot page, a corresponding first copy-on-write file page is created in the first copy-on-write file, data of the first database snapshot page is copied to the first copy-on-write file page, and the source database node performs a write operation in the first copy-on-write file, where the first database snapshot page is one or more of the database snapshot pages.

[0084] If the write operation of the transaction resumed by the source database node is not the first write operation on the first database snapshot page, the source database node performs the write operation in the first copy-on-write file.

[0085] For a write operation of a destination database node, the destination database node uses the database snapshot page to execute the received transaction based on a copy-on-write mode, including:

[0086] If the write operation of the transaction to be executed by the destination database node is the first write operation to the second database file, creating a corresponding second copy-on-write file for the second database file, where the second database file is the database file of the destination database node;

[0087] If the write operation of the transaction to be executed by the destination database node is the first write operation on the second database snapshot page, create a corresponding second copy-on-write file page in the second copy-on-write file, copy data of the second database snapshot page to the second copy-on-write file page, where the second database snapshot page is one or more of the database snapshot pages, and the destination database node performs the write operation in the second copy-on-write file;

[0088] If the write operation of the transaction to be executed by the destination database node is not the first write operation on the second database snapshot page, the destination database node performs the write operation in the second copy-on-write file.

[0089] Accordingly, in order to improve the reading and writing efficiency, the method further includes:

[0090] After the source database node creates the first copy-on-write file page, writing the first database snapshot page corresponding to the first copy-on-write file page back to the shared storage area, and writing the first copy-on-write file page into a local cache pool;

[0091] After the destination database node creates the second copy-on-write file page, the second database snapshot page corresponding to the second copy-on-write file page is deleted, and the second copy-on-write file page is written to the local cache pool.

[0092] See also Figure 3 The write operation diagram of the copy-on-write mode is shown. According to the write operation process of the source database node and the destination database node described above, the source database node and the destination database node adopt the same strategy: when a transaction completes the snapshot operation on the source database node and wants to modify a database file for the first time, a copy-on-write file (i.e., COW file) and a copy-on-write index file (i.e., COW index file) are created for the database file. The first modification of the database snapshot page will trigger the allocation of a new physical page (i.e., COW file page) from the corresponding copy-on-write file, and the contents of the database snapshot page are copied to the physical page of the copy-on-write file before the modification operation is performed. The copy-on-write file is stored in a shared storage area. Specifically, the shared storage area may include: a storage area for storing index, heap, and transaction status information, a copy-on-write file storage area, a database snapshot page storage area, etc. The source database node and the destination database node can add different prefixes to the file names to distinguish them in the shared storage area. For example, the source database node may add a prefix "src" before the copied file and the corresponding index file during the first write, and the destination database node may add a prefix "dest" before the copied file and the corresponding index file during the second write.

[0093] During a write operation on either the source or destination database node, the relevant database snapshot page in the local cache pool can be eliminated and replaced with the corresponding copy-on-write file page. Furthermore, for the source database node, if the database snapshot page associated with the write operation is dirty, the dirty page can also be persistently written to the shared storage area. To further improve read and write efficiency, the page numbers of the database snapshot page and the copy-on-write file page are recorded in the corresponding copy-on-write index file. This allows subsequent reads and writes to be routed to the copy-on-write file page.

[0094] It can be seen from the above process that when performing a write operation in the copy-on-write mode (such as inserting a new record in a table, etc.), when the database table / index file may need to allocate new page space, it is allocated directly from the copy-on-write file, rather than from the original table / index file. In addition, the copy-on-write process of the source database node can be recorded in the copy-on-write log, and the copy-on-write process of the destination database node can be recorded in the database log in the shared storage area, so as to ensure fault recovery. Specifically, for example, the database log can record: the allocation operation content of the new COW page, the insertion operation content of the COW index, the operation content of creating a COW file, the operation content of the index and the subsequent modification content of the COW page, etc.

[0095] Furthermore, since dirty pages may exist in the database snapshot pages in the cache pool when the source database node performs a snapshot operation, the dirty pages of the source database node must be flushed before the dirty pages of the destination database node can be flushed. This ensures that the dirty flush operation of the destination database node can obtain consistent database data. Based on this, the method may further include: after completing the snapshot operation, the source database node performs a dirty flush operation, wherein the dirty flush operation flushes the dirty data of the source database to the shared storage area; and after the source database node completes the dirty flush operation, the destination database node performs a first dirty flush operation.

[0096] Specifically, when the source database node performs a dirty flush operation, the dirty pages of the database snapshot pages in the cache pool are flushed to the snapshot area in the shared storage area, specifically, written to the heap table and index files. When the destination database node performs a dirty flush operation, the database snapshot pages in the cache pool can be directly discarded. However, dirty data in the copy-on-write files in the cache pools of both the source and destination database nodes must be flushed to the corresponding copy-on-write files in the shared storage area during the dirty flush operation. Therefore, the source and destination database nodes each perform dirty flush operations, enabling the database snapshot pages and copy-on-write pages in the shared storage area to support fault recovery. For example, after the source database node completes the snapshot operation and creates the snapshot point, the source database node initiates a final dirty flush operation, writing the dirty data pages of the database snapshot pages back to the shared storage area and finally creating a checkpoint. On the destination database node, when the database snapshot pages in the cache pool are replaced with copy-on-write file pages, the replaced database snapshot pages are discarded. After the source database node completes the final dirty flush operation, the destination database node takes over and performs subsequent dirty flush operations.

[0097] When the source database node flushes the database log records in the memory to the shared storage area, the latest database log record sequence number in the flushed database log records can be used as the snapshot point.

[0098] Step 210: The source database node sends the modified content to the destination database node.

[0099] The modification content is the modification content corresponding to the write operation performed by the source database node after the snapshot operation. For example, the modification content may include one or more logical modification records, each of which includes: a modification record identifier, a modification operation type, and a corresponding modification value.

[0100] The source database node may record the modified content in any manner. For example, in one embodiment, the source database node may record the modified content corresponding to the write operation in a file of any type while executing the write operation. In another embodiment, the modified content corresponding to the write operation may be recorded in a copy-on-write log based on the copy-on-write mode.

[0101] Step 212: The destination database node performs logical playback using the modified content.

[0102] The logic playback refers to the destination database node obtaining the corresponding database snapshot page according to the snapshot timestamp of the transaction to which the modified content belongs, and running the modification logic corresponding to the modified content, so that the destination database node obtains the same output result as when the source database node runs the transaction.

[0103] In order to avoid log write conflicts and support fault recovery, in one or more embodiments of the present specification, after the source database node suspends the transaction being executed and before resuming the execution of the transaction, the source database node can write the database log in the memory to the shared storage area. In this way, when the destination database node executes a new transaction after the snapshot, the modified content of the new transaction can be written to the database log in the shared storage area. The destination database node can obtain the complete database log before and after the snapshot, ensuring that the database log can support fault recovery. Therefore, the destination database node records the modified content corresponding to the executed write operation to the database log in the shared storage area. In addition, in order to avoid log write conflicts between the source and destination database nodes, the source database node can record the modified content corresponding to the write operation executed after resuming the execution of the transaction to the write-time copy log. Accordingly, the source database node sending the modified content to the destination database node can include: the source database node reads the write-time copy log, parses the modified content of each log item in the write-time copy log, and sends it to the destination database node.

[0104] Specifically, see Figure 4 The log processing process during the database migration process is shown in Figure 1. Figure 4 As shown, the local cache pool of the source database node may include: a first write-time copy file page cache pool, a database snapshot page cache pool, and a log cache pool. The local cache pool of the destination database node may include: a second write-time copy file page cache pool, a database snapshot page cache pool, and a log cache pool. When the source database node performs a snapshot operation, the transaction of the source database node is suspended, and the database log in the source database node memory is flushed to the shared storage area. At this time, the end position of the database log is recorded as the snapshot point. After the snapshot operation is completed, the source database node enters the write-time copy mode and creates a write-time copy log ( Figure 4 In the area of ​​the copy-on-write log (cow WAL log is used to illustrate the copy-on-write log), the source database node executes transactions based on the copy-on-write mode. When the source database node performs a write operation, a first copy-on-write file page is generated based on the database snapshot page, and the first copy-on-write file page is cached in the first copy-on-write file page cache pool. When the source database node performs a read operation, the read database snapshot page is cached in the database snapshot page cache pool. The modified content corresponding to the write operation performed by the source database node is recorded in the copy-on-write log of the log cache pool. It can be understood that the source database node can flush the dirty data of the database snapshot page in the cache, the dirty data of the copy-on-write file page, and the dirty data of the copy-on-write log to the shared storage area under any circumstances (such as flushing dirty data, writing cache and writing persistent storage at the same time, etc.). Among them, the database log written to the shared storage area is at the snapshot point ( Figure 4After the snapshot point (snap_lsn; is used to indicate the snapshot point), no modifications will be made to the source database node. The destination database node executes transactions based on the copy-on-write mode. When the destination database node performs a write operation, a second copy-on-write file page is generated based on the database snapshot page, and the second copy-on-write file page is cached in the second copy-on-write file page cache pool. When the destination database node performs a read operation, the read database snapshot page is cached in the database snapshot page cache pool. The modifications corresponding to the write operation performed by the destination database node are recorded in the log cache pool and synchronized to the database log in the shared storage area as needed. This is equivalent to the write log operation of the database log, which is executed by switching from the original source database node to the destination database node. In the database log, the log records after the snapshot point are composed of the modifications made by the destination database node.

[0105] According to the above embodiment, the source database node can also start a parsing process to use the parsing process to parse the modification content from the copy-on-write log and send it to the destination database node for logical playback. The source database node's logical log process is as follows: the parsing process reads each log entry in the copy-on-write log, parses the table tuple modified by each log entry, modifies the operation type (such as insert / delete / update / lock), modifies the value (the inserted value or updated tuple column and value), generates a logical modification record (i.e., the modification content), and then sends the logical modification record to the destination database node for playback.

[0106] In addition, in order for the destination database node to quickly locate the modified tup le during logical replay, one method is to use the corresponding primary key of each table in the database. When the source database node records the modified content in the write-time copy log, it also records the primary key of the table corresponding to the modified content. In this way, during logical replay, the destination database node can quickly locate the tup le to be modified through the primary key information in the modified content. Another method is to add a hidden row number column to the table. In this way, each tup le in the table will generate an incremental and unique RID when it is generated, and build an RID index. When the source database node records the modified content in the write-time copy log, it also records the RID of the modified tup le. In this way, during logical replay, the destination database node can quickly locate the tup le to be modified based on the RID index.

[0107] In the methods provided in the embodiments of this specification, to ensure that the destination database node obtains globally consistent database data, the source database node's transaction is committed using the 2PC (two-phase commit protocol, where P stands for Prepare and C stands for Commit) protocol after a snapshot operation. After the Prepare phase is completed, the destination database node must wait for the playback result. If there are no write-write conflicts and the playback is successful, the destination database node will return the playback result with a commit timestamp. The source database node then commits the source transaction using the same commit timestamp. Otherwise, the transaction is rolled back.

[0108] Specifically, in one or more embodiments of this specification, the first transaction is a two-phase transaction. The source database node reads the copy-on-write log, parses the modified content of each log entry in the copy-on-write log, and sends it to the destination database node, including:

[0109] The source database node creates a cache queue for each transaction;

[0110] The source database node reads log entries in the write-time copy log in order from earliest to latest;

[0111] The source database node parses the modified content of the log item;

[0112] The source database node places the parsed modified content into the cache queue of the corresponding transaction;

[0113] In response to any transaction being in the preparation phase, the source database node sends the modified content cached in the cache queue of the transaction to the destination database node.

[0114] In the above embodiment, the parsing process of the source database node continuously parses the write-time copy log in sequence, and creates a cache queue for each source transaction (transaction of the source data node) corresponding to the modified content, which is used to cache the parsed logical modification records of the transaction. In this way, when the source database node responds to any source transaction in the preparation stage, it sends all logical modification records in the cache queue of the transaction to the destination database node for logical playback, and then deletes the cache queue of the transaction.

[0115] In order to avoid write-write conflicts on the destination database node during logical playback, in one or more embodiments of this specification, the destination database node performs logical playback using the modified content, including:

[0116] The destination database node initiates a corresponding shadow transaction for the transaction to which the modified content belongs, wherein the shadow transaction includes: a read operation on a database snapshot page related to the modified content; write conflict detection, lock conflict detection, and constraint condition check on data read by the read operation; and, if the write conflict detection, lock conflict detection, and constraint condition check all pass, a logical playback operation using the modified content. The read operation uses the same snapshot timestamp as that of the transaction to which the modified content belongs to read the database snapshot page;

[0117] If any one of the write conflict detection, lock conflict detection and constraint condition check fails, the destination database node terminates execution of the shadow transaction;

[0118] If the shadow transaction successfully completes the logic replay operation, the destination database node assigns a commit timestamp to the shadow transaction, commits the shadow transaction, and sends the commit timestamp to the source database node, so that the source database node commits the corresponding transaction using the commit timestamp;

[0119] If the logical playback of the shadow transaction fails or is aborted, the destination database node returns failure information to the source database node, causing the source database node to roll back the corresponding transaction.

[0120] The shadow transaction refers to a transaction that uses the same snapshot timestamp as the corresponding source transaction to perform the same modification content and produces the same result on the database as the source transaction.

[0121] For example, according to the above embodiment, the destination database node can initiate an apply process to perform logical replay. The apply process is a blocking process. First, the main process starts running, encounters a child process, switches to the child process, waits for the child process to complete, and then switches to another child process until all child processes have completed. Specifically, before replaying the modified content set of any source transaction, the apply process initiates a shadow transaction for it. A source transaction refers to the transaction that the source database node resumes executing after a snapshot operation. For each modification operation record of the source transaction, the apply process performs a snapshot read using the same snapshot timestamp as the source transaction, performing write-write conflict detection, lock conflict detection, and constraint condition checks. Specifically, the apply process locates the tuplet version chain to be modified based on the primary key or RID in the modification operation record. It then reads the latest version submitted before the snapshot timestamp to perform write-write conflict detection. If a new version exists after the latest version submitted before the snapshot timestamp, or if the version is marked for deletion, this indicates that the tuplet has been modified by a concurrent transaction on the destination database node, resulting in a concurrent write-write conflict and the need to abort the shadow transaction and the corresponding source transaction.

[0122] Based on the above embodiment, the modification content sent by the source database node to the destination database node may also include locking operation information. Accordingly, the source database node records the locking operation information performed during the transaction execution process in the write-time copy log. In this way, when the source database node parses the write-time copy log, it can parse the locking operation information and then send the locking operation information to the destination database node, so that the destination database node performs the corresponding locking playback when executing the logical playback. For example, the lock information of the tuple can be embedded in the tuple header and the combined transaction ID log. The destination database node can obtain the lock added by the source database node before the snapshot by reading the snapshot data of the heap table and the combined transaction ID log. After the snapshot, the source database node writes the locking operation information to the write-time copy log. The destination database node can perform lock playback by obtaining the locking operation information sent by the source database node.

[0123] For example, for explicit locks (such as SQL statements used to read data, such as "Select for update / share"), the source database node can synchronize the locking operation information of the explicit lock in the source transaction to the destination database node for playback. During the logical playback process, the destination database node can locate the tuple to be locked based on the primary key or RID in the locking operation information and then perform the locking operation. If there is a lock conflict with concurrent writes on the destination database node, the shadow transaction and the corresponding source transaction are aborted.

[0124] For all logical modification records, if there are no write-write conflicts and the constraint check passes, the apply process assigns a commit timestamp, commits the shadow transaction, and returns the commit timestamp to the source database node. Otherwise, the shadow transaction replay fails and is aborted, with a failure message returned to the source database node. Constraint checks are defined based on database scenarios, such as uniqueness constraints. Upon receiving a replay success message, the source database node commits the source transaction with the same commit timestamp. If a replay failure message is received, the source transaction is rolled back.

[0125] Furthermore, the write operations generated by the destination database node during logical playback can also be executed in a copy-on-write mode by copying the relevant database snapshot page to the second copy-on-write file. Specifically, the first time the destination database node modifies a database snapshot page, it generates a corresponding second copy-on-write file page. Subsequent modifications are performed directly on the second copy-on-write file page. To improve playback efficiency, the destination database node can initiate multiple threads for parallel playback, thereby accelerating playback speed. For example, a transaction-level parallel playback algorithm can be employed.

[0126] Step 214: When the source database node determines that the logic playback of the destination database node is successfully completed, the source database node commits the transaction corresponding to the modified content.

[0127] As can be seen, according to the method provided in the embodiments of this specification, when a database service needs to be migrated from a source database node to a destination database node, the source database node will suspend the currently executing transaction, perform a snapshot operation, and obtain a database snapshot page. After the snapshot operation is completed, not only can the source database node resume executing transactions based on the copy-on-write mode, but transactions received after the snapshot operation can also be routed to the destination database node, causing the destination database node to execute transactions based on the copy-on-write mode. In this way, the source database node and the destination database node each execute transactions before and after the snapshot without interrupting database service. Furthermore, the source database node sends the modified content of the write operation performed after the snapshot operation to the destination database node, causing the destination database node to perform logical replay using the modified content. This not only ensures that the destination database node obtains globally consistent database data, but also reduces the amount of data transmission and resource overhead caused by the migration. Subsequently, when the logical replay of the destination database node is successfully completed, the source database node commits the transactions running on it, completing the migration of the database service. This implements a database migration method that does not require interruption of database service during the migration process and has low resource overhead.

[0128] In one or more embodiments of this specification, the above embodiments are combined to achieve the following Figure 5 The online migration architecture based on shared storage space is shown in Figure 1. Figure 5 As shown, in the online migration architecture, it includes: a proxy server, a source database node and a destination database node. Among them, the proxy server is used to receive transactions and route the transactions to the source database node ( Figure 5 In the example, Src RW node is used to indicate the source database node) or the destination database node ( Figure 5 (Dest RW node is used to represent the destination database node.) The source database node performs a snapshot operation when migration is required. After the snapshot operation is completed, both the source and destination database nodes simultaneously execute the old and new transactions using the copy-on-write mode. The source and destination database nodes are database instances that store database data in a shared storage area. When executing a transaction using the copy-on-write mode, the source database node records the changes corresponding to the write operation in the copy-on-write log. The changes are then parsed and sent to the destination database node for logical playback.

[0129] In order to make Figure 5 The migration architecture shown is easier to understand. Figure 6 , further describes the signaling interaction process of the database migration method. Figure 6 A signaling interaction diagram of a database migration method provided in an embodiment of this specification is shown, which specifically includes the following steps.

[0130] Step 602: Start a database instance as a destination database node on the destination machine based on the same database directory as the source database node.

[0131] Step 604: The source database node determines whether there is an atomic modification operation being executed.

[0132] Step 606: If there is an atomic modification operation being executed, wait for the completion of the atomic modification operation and enter the step of suspending the transaction being executed by the source database node.

[0133] Step 608: If there is no atomic modification operation being executed, the source database node suspends the transaction being executed.

[0134] Step 610: The source database node suspends the currently executing transaction, performs a snapshot operation, and obtains a database snapshot page.

[0135] During the period from snapshot start to snapshot end, the proxy server suspends incoming transactions.

[0136] Step 612: The source database node writes the database log in the memory to the shared storage area.

[0137] Step 614: After the source database node completes the snapshot operation, the source database node resumes executing transactions.

[0138] After the snapshot is completed, the proxy server routes the newly arrived transactions to the destination database node.

[0139] Step 616: The destination data node receives the routed transaction.

[0140] Step 618: After completing the snapshot operation, the source database node performs a dirty flush operation, which flushes the dirty data of the source database to the shared storage area.

[0141] Step 620: For a write operation, the source database node performs the write operation in a first copy-on-write file by copying the relevant database snapshot page based on the copy-on-write mode.

[0142] Step 622: The source database node records the modified content corresponding to the write operation performed after the transaction is resumed in the copy-on-write log.

[0143] Step 624: For the write operation, the destination database node performs the write operation in the second copy-on-write file by copying the relevant database snapshot page based on the copy-on-write mode.

[0144] Step 626: The destination database node records the modified content corresponding to the executed write operation into the database log in the shared storage area.

[0145] Step 628: For read operations, the source database node first determines whether the data to be read is in the local cache pool. If so, the data to be read is obtained from the local cache pool. If not, it determines whether the data to be read is in the first write-time copy file. If so, the data to be read is obtained from the first write-time copy file and placed in the local cache pool. Otherwise, the data to be read is obtained from the database snapshot page stored in the shared storage area and placed in the local cache pool.

[0146] Step 630: For a read operation, the destination database node determines whether the data to be read is in the local cache pool. If so, the destination database node obtains the data to be read from the local cache pool. If not, the destination database node determines whether the data to be read is in the second copy-on-write file. If so, the destination database node obtains the data to be read from the second copy-on-write file and places it in the local cache pool. Otherwise, the destination database node obtains the data to be read from the database snapshot page stored in the source database node or the shared storage area and places it in the local cache pool.

[0147] Step 632: The source database node reads the copy-on-write log, parses the modified content of each log entry in the copy-on-write log, and sends it to the destination database node.

[0148] Step 634: The destination database node performs logical playback using the received modified content.

[0149] Step 636: When the source database node determines that the logic playback of the destination database node is successfully completed, the source database node commits the transaction corresponding to the modified content.

[0150] It can be seen from the above processing that when database migration is required, the source database node performs a snapshot operation. After the snapshot operation is completed, the write operations performed by the source database node and the destination database node are respectively executed in their respective COW files. The modification content corresponding to the write operation of the source database node is recorded in the write-time copy log, and the modification content corresponding to the write operation of the destination database node is recorded in the database log, thereby avoiding write log conflicts. Moreover, when the destination database node needs to access the database snapshot page, it obtains the corresponding database snapshot page from the source database node through the high-speed network, thereby providing a strongly consistent destination database node. In addition, the source database node records the modification content corresponding to the write operation executed after the reply transaction is executed in the write-time copy log, and then parses the write-time copy log to send the modification content to the destination database node for logical playback, so that the destination database node obtains a globally consistent view, realizing a database migration method that does not require interruption of database services during the migration process and has low resource overhead.

[0151] In one or more embodiments of this specification, in order to ensure consistency of transaction status information, the method further includes:

[0152] The source database node modifies the transaction state information in the third copy-on-write file by copying relevant data pages in the transaction state maintenance file based on the copy-on-write mode, wherein the transaction state maintenance file is used to maintain the transaction state information and is stored in the shared storage area;

[0153] The destination database node modifies the transaction status information in the fourth copy-on-write file by copying the relevant data pages in the transaction status maintenance file based on the copy-on-write mode.

[0154] In the above embodiment, transaction status information is maintained by establishing a data structure called a "transaction status maintenance file" and stored in shared storage. For example, in some types of databases, transaction status information may include a transaction commit status (timestamp) storage log, a sub-transaction status storage log, and a multi-ixact id log that supports share-lock. Among them, share-lock represents a shared lock, and multi-ixact id log represents a combined transaction ID log. These transaction status information are stored in the shared storage area in the form of files, divided and managed according to physical pages, and a page cache is established in the memory to accelerate page read and write access. After the snapshot operation, for the modification of transaction status information, the source database node and the destination database node can adopt the same write-time copy mode as for data pages.

[0155] In one or more embodiments of this specification, in order to ensure transaction consistency and session consistency, the method further includes:

[0156] After the snapshot operation starts, the source database node sends the next available transaction unique identifier and transaction global variable information to the destination database node, so that the destination database node maintains the available transaction unique identifier and transaction global variable information;

[0157] When the source database node needs an available transaction unique identifier, the destination database node allocates an available transaction unique identifier to the source database node.

[0158] In the above embodiment, the source database node sends the next available transaction unique identifier and other transaction global variable information to the destination database node for maintenance. During the bilateral operation between the source database node and the destination database node, the destination database node assigns a transaction unique identifier to the source database node, thereby ensuring that the transaction unique identifier remains consistent and unified in the source / destination database nodes, thereby ensuring the consistency of the transaction.

[0159] Furthermore, the destination database node's list of running transactions also needs to consider transactions currently running on the source database node. Specifically, during a snapshot operation, the source database node sends its list of running transactions to the destination database node. Accordingly, before terminating a transaction on the source database node, it must be removed from the destination database node's list of running transactions to ensure global consistency of transactions maintained by the destination database node. As will be appreciated, if a transaction is not removed, the results of the terminated transaction will become invisible, leading to data inconsistency and corruption.

[0160] In one or more embodiments of this specification, routing the transaction received after the snapshot operation to the destination database node includes:

[0161] The proxy server suspends the transaction received during the execution of the snapshot operation on the source database node;

[0162] When the source database node finishes executing the snapshot operation, the proxy server assigns a start timestamp to the suspended transaction;

[0163] The proxy server routes the transaction assigned with the start timestamp to the destination database node.

[0164] In the above embodiment, during the snapshot operation performed by the source database node, the proxy server will suspend the newly arrived transaction, wait for the snapshot to end, and then assign a start timestamp to the newly arrived transaction, and then route it to the destination database node. Based on the principle of timestamp increment, it can be ensured that the start timestamp of the new transaction routed to the destination database node is greater than (or understood to be later than) the start timestamp of the old transaction routed to the source database node. Since the commit timestamp is greater than the start timestamp, the start timestamp of the old transaction executed by the source database node must be less than the commit timestamp of the new transaction executed by the destination database node. The modification of the new transaction executed by the destination database node is not visible to the old transaction executed by the source database node. Therefore, under the dual execution model of the source database node and the destination database node, according to the method provided in the embodiment of this specification, it is only necessary to synchronize the modified content of the transaction of the source database node to the destination database node, and perform logical replay after detection such as write-write conflict detection and lock conflict detection to ensure snapshot isolation. Among them, the start timestamp and commit timestamp of each transaction can be assigned by the proxy server.

[0165] In one or more embodiments of the present specification, in order not to affect the execution of transactions of the source database node when the MVCC space of the destination database node is reclaimed, the method further includes: when the destination database node reclaims memory space based on the multi-version transaction concurrency control mechanism, if the database snapshot page of the version to be reclaimed is currently being used by the transaction of the source database node, then the recovery of the database snapshot page of the version to be reclaimed is abandoned. It can be understood that a timestamp is obtained at the beginning of any transaction, which means that the results before this time are visible to the transaction. When this embodiment reclaims the old version of data, it is necessary to consider the transactions running on the source database node. If the timestamp of a transaction is relatively small, that is, smaller than the timestamp of the old version of the data, then the corresponding version of the data cannot be recovered. Therefore, in this embodiment, when the destination database node performs multi-version recovery, it will also consider the transactions of the source database node and retain a visible version for each transaction of the source database node.

[0166] In one or more embodiments of the present specification, when the last flush operation of the source database node is completed and all transactions of the source database node are completed, the write operation of the destination database node exits the copy-on-write mode. After exiting the copy-on-write mode, if the read operation of the destination database node is a read operation of data in the second copy-on-write file, the read operation can continue to be performed on the second copy-on-write file. In order to reduce the overhead caused by searching the copy-on-write file, in one or more embodiments of the present specification, the destination database node can also write the data in the second copy-on-write file back to the corresponding database snapshot page, and record the operation of writing the data in the second copy-on-write file back to the corresponding database snapshot page in the database log.

[0167] It is understood that according to the method provided in the embodiments of this specification, all transactions of the source database node are completed, and the last flush operation of the source database node is completed, which means that the migration of the database service from the source database node to the destination database node is completed. Therefore, in the above embodiment, after all transactions of the source database node are completed, the destination database node exits the copy-on-write mode and enters the normal read-write mode, that is, modifying the database snapshot page will no longer generate a copy-on-write file page. Moreover, the destination database node merges the second copy-on-write file by writing it back to the corresponding database snapshot page, reducing the additional overhead caused by searching for the copy-on-write file page. To improve the efficiency of merging, in one or more embodiments of this specification, the destination database node can initiate one or more merge processes to merge and clean up each copy-on-write file page. In addition, when each copy-on-write file page is written back to its corresponding database snapshot page, the corresponding COW index entry can also be deleted. During the merging process, this embodiment can add page locks and / or buffer locks to ensure data consistency.

[0168] It should be noted that during the database migration process, failures can occur in the source database node, the destination database node, or both. If the failure occurs before the snapshot operation is complete, the source database node can be restored based on the database log. If the failure occurs after the snapshot operation is complete, the destination database node can be restored based on the database log.

[0169] Corresponding to the above method embodiments, this specification also provides a database migration method applied to a source database node. Figure 7 FIG1 shows a flow chart of a database migration method applied to a source database node provided by an embodiment of this specification. Figure 7 As shown, the method includes:

[0170] Step 702: Suspend the currently executing transaction, perform a snapshot operation, and obtain a database snapshot page.

[0171] Step 704: After the snapshot operation is completed, the database snapshot page is used to resume executing transactions based on the copy-on-write mode.

[0172] The snapshot operation is used to route transactions received after the snapshot operation to a destination database node.

[0173] The database snapshot page is further configured to enable a destination database node to execute the received transaction using the database snapshot page based on a copy-on-write mode after receiving the transaction, wherein the destination database node and the source database node are database instances based on a shared storage area for storing database data;

[0174] Step 706: Send the modified content of the write operation executed after the snapshot operation to the destination database node, so that the destination database node performs logical playback using the modified content.

[0175] Step 708: When it is determined that the logic playback of the destination database node is successfully completed, the transaction corresponding to the modified content is committed.

[0176] According to the above method, when a database service needs to be migrated from a source database node to a destination database node, the source database node suspends the currently executing transaction, performs a snapshot operation, and obtains a database snapshot page. After the snapshot operation is completed, not only can the source database node resume executing transactions based on a copy-on-write mode, but transactions received after the snapshot operation can also be routed to the destination database node, causing the destination database node to execute transactions based on a copy-on-write mode. In this way, the source database node and the destination database node each execute transactions before and after the snapshot without interrupting database service. Furthermore, the source database node sends the modified content of the write operation performed after the snapshot operation to the destination database node, causing the destination database node to perform logical replay using the modified content. This not only ensures that the destination database node obtains globally consistent database data, but also reduces the amount of data transmission and resource overhead associated with the migration. Furthermore, upon successful completion of the logical replay on the destination database node, the source database node commits the transactions running on it, completing the migration of the database service. This implements a database migration method that does not require interruption of database service during the migration process and has low resource overhead.

[0177] The above is a schematic diagram of the database migration method applied to the source database node according to this embodiment. It should be noted that the technical solution of this database migration method applied to the source database node shares the same concept as the technical solution of the database migration method described in any of the aforementioned embodiments. For details not described in detail in the technical solution of the database migration method applied to the source database node, please refer to the description of the technical solution of the aforementioned database migration method.

[0178] Corresponding to the above method embodiment, this specification also provides an embodiment of a database migration device configured on a source database node. Figure 8 FIG1 shows a schematic diagram of the structure of a database migration device configured on a source database node provided by an embodiment of this specification. Figure 8 As shown, the device includes:

[0179] The snapshot module 802 is configured to suspend the currently executing transaction, perform a snapshot operation, and obtain a database snapshot page.

[0180] The first transaction execution module 804 is configured to resume executing transactions based on the copy-on-write mode using the database snapshot page after completing the snapshot operation, wherein the snapshot operation is used to route transactions received after the snapshot operation to the destination database node.

[0181] The database snapshot page is further used to enable the destination database node to execute the received transaction based on the copy-on-write mode using the database snapshot page after receiving the transaction. The destination database node and the source database node are database instances that store database data based on a shared storage area.

[0182] The modification sending module 806 is configured to send the modified content of the write operation executed after the snapshot operation to the destination database node, so that the destination database node performs logic playback using the modified content.

[0183] The commit module 808 is configured to commit the transaction corresponding to the modified content when it is determined that the logic playback of the destination database node is successfully completed.

[0184] The above is a schematic diagram of the database migration apparatus configured at the source database node according to this embodiment. It should be noted that the technical solution of this database migration apparatus configured at the source database node shares the same concept as the technical solution of the database migration method described in any of the aforementioned embodiments. For details not described in detail in the technical solution of the database migration apparatus configured at the source database node, please refer to the description of the technical solution of the aforementioned database migration method.

[0185] Corresponding to the above method embodiments, this specification also provides a database migration method applied to a destination database node. Figure 9 A flowchart of a database migration method configured on a destination database node provided by one embodiment of this specification is shown.

[0186] like Figure 9 As shown, the method includes:

[0187] Step 902: Receive a transaction. The transaction is a transaction that is routed to the destination database node after the source database node performs a snapshot operation and obtains a database snapshot page. The destination database node and the source database node are database instances that store database data based on a shared storage area.

[0188] Step 904: Execute the received transaction using the database snapshot page based on the copy-on-write mode. The database snapshot page is also used to enable the source database node to resume executing the transaction using the database snapshot page based on the copy-on-write mode.

[0189] Step 906: Obtain modification content corresponding to the write operation performed by the source database node after the snapshot operation.

[0190] Step 908: Perform logic playback using the modified content, so that the source database node commits the transaction corresponding to the modified content when determining that the logic playback of the destination database node is successfully completed.

[0191] The above is a schematic diagram of the database migration method applied to the destination database node according to this embodiment. It should be noted that the technical solution of this database migration method applied to the destination database node shares the same concept as the technical solution of the database migration method described in any of the aforementioned embodiments. For details not described in detail in the technical solution of the database migration method applied to the destination database node, please refer to the description of the technical solution of the aforementioned database migration method.

[0192] Corresponding to the above method embodiment, this specification also provides an embodiment of a database migration device configured on a target database node. Figure 10 FIG. 1 shows a schematic diagram of the structure of a database migration device configured at a target database node provided by an embodiment of this specification. Figure 10 As shown, the device includes:

[0193] The receiving module 1002 is configured to receive a transaction, where the source database node performs a snapshot operation, obtains a database snapshot page, and routes the transaction to the destination database node. The destination database node and the source database node are database instances that store database data based on a shared storage area.

[0194] The second transaction execution module 1004 is configured to execute the received transaction based on the copy-on-write mode using the database snapshot page. The database snapshot page is also used to enable the source database node to resume executing the transaction based on the copy-on-write mode using the database snapshot page.

[0195] The modification acquisition module 1006 is configured to acquire the modification content corresponding to the write operation performed by the source database node after the snapshot operation;

[0196] The logic playback module 1008 is configured to perform logic playback using the modified content, so that the source database node commits the transaction corresponding to the modified content when determining that the logic playback of the destination database node is successfully completed.

[0197] The above is a schematic diagram of the database migration apparatus configured at the destination database node according to this embodiment. It should be noted that the technical solution of this database migration apparatus configured at the destination database node shares the same concept as the technical solution of the database migration method described in any of the aforementioned embodiments. For details not described in detail in the technical solution of the database migration apparatus configured at the destination database node, please refer to the description of the technical solution of the aforementioned database migration method.

[0198] Corresponding to the above method embodiments, this specification also provides a database migration system. Figure 11 FIG. 1 shows a schematic diagram of the structure of database migration coordination provided by an embodiment of this specification. Figure 11 As shown, the system includes:

[0199] A source data database node 1102 implemented by the database migration method described in any embodiment of this specification, and a destination database node 1104 implemented by the database migration method described in any embodiment of this specification.

[0200] Figure 12 The following is a block diagram of a computing device 1200 according to one embodiment of the present disclosure. Components of the computing device 1200 include, but are not limited to, a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and a database 1250 is used to store data.

[0201] The computing device 500 also includes an access device 1240 that enables the computing device 1200 to communicate via one or more networks 1260. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1240 may include one or more of any type of network interface, wired or wireless (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0202] In one embodiment of the present specification, the above components of the computing device 1200 and Figure 12 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 12 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0203] Computing device 1200 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 500 may also be a mobile or stationary server.

[0204] The processor 1220 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the source database node and / or the destination database node in the above-mentioned database migration system.

[0205] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of this computing device is based on the same concept as the technical solution of the aforementioned database migration system. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the aforementioned database migration system.

[0206] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned database migration method.

[0207] The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium is based on the same concept as the technical solution of the aforementioned database migration method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the aforementioned database migration method.

[0208] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned database migration method.

[0209] The above is a schematic diagram of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the aforementioned database migration method are based on the same concept. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the aforementioned database migration method.

[0210] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0211] The computer instructions include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0212] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0213] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0214] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A database migration method, comprising: The source database node suspends the executing transaction, performs a snapshot operation, and obtains the database snapshot page; After completing the snapshot operation, the source database node uses the database snapshot page to resume executing transactions based on a copy-on-write mode; Routing transactions received after the snapshot operation to a destination database node, where the destination database node and the source database node are database instances storing database data based on a shared storage area; The destination database node executes the received transaction based on a copy-on-write mode using the database snapshot page; The source database node sends the modified content to the destination database node, so that the destination database node performs logic playback using the modified content, where the modified content corresponds to the write operation performed by the source database node after the snapshot operation; The source database node commits the transaction corresponding to the modified content when determining that the logic replay of the destination database node is successfully completed; The destination database node performs logical playback using the modified content, including: The destination database node initiates a corresponding shadow transaction for the transaction to which the modified content belongs, wherein the shadow transaction includes: a read operation on a database snapshot page related to the modified content; write conflict detection, lock conflict detection, and constraint condition check on data read by the read operation; and, if the write conflict detection, lock conflict detection, and constraint condition check all pass, a logical playback operation using the modified content. The read operation uses the same snapshot timestamp as that of the transaction to which the modified content belongs to read the database snapshot page; If any one of the write conflict detection, lock conflict detection and constraint condition check fails, the destination database node terminates execution of the shadow transaction; If the shadow transaction successfully completes the logic replay operation, the destination database node assigns a commit timestamp to the shadow transaction, commits the shadow transaction, and sends the commit timestamp to the source database node, so that the source database node commits the corresponding transaction using the commit timestamp; If the logical playback of the shadow transaction fails or is aborted, the destination database node returns failure information to the source database node, causing the source database node to roll back the corresponding transaction.

2. The method according to claim 1, further comprising: After the source database node suspends the transaction and before resuming the transaction, the source database node writes the database log in the memory to the shared storage area; The source database node records the modified content corresponding to the write operation performed after the transaction is resumed in the copy-on-write log; The destination database node records the modified content corresponding to the executed write operation in the database log in the shared storage area; The source database node sends the modified content to the destination database node, including: The source database node reads the copy-on-write log, parses the modified content of each log item in the copy-on-write log, and sends the modified content to the destination database node.

3. The method according to claim 1, before the source database node suspends the currently executing transaction and performs the snapshot operation, further comprising: The source database node determines whether there is an atomic modification operation being executed; If there is an atomic modification operation being executed, wait for the atomic modification operation to be completed, and then enter the step of suspending the transaction being executed by the source database node; If there is no atomic modification operation being executed, the process proceeds to a step in which the source database node suspends the transaction being executed.

4. The method according to claim 1, further comprising: After completing the snapshot operation, the source database node performs a dirty flushing operation, where the dirty data of the source database is flushed to the shared storage area. After the source database node completes the dirty flushing operation, the destination database node performs a first dirty flushing operation.

5. The method according to claim 1, wherein the source database node uses the database snapshot page to resume executing transactions based on a copy-on-write mode, comprising: If the write operation of the transaction resumed by the source database node is the first write operation on the first database file after the snapshot operation, create a corresponding first copy-on-write file for the first database file, where the first database file is the database file of the source database node; If the write operation of the transaction resumed by the source database node is the first write operation on a first database snapshot page, a corresponding first copy-on-write file page is created in the first copy-on-write file, data of the first database snapshot page is copied to the first copy-on-write file page, and the source database node performs a write operation in the first copy-on-write file, where the first database snapshot page is one or more of the database snapshot pages. If the write operation of the transaction resumed by the source database node is not the first write operation on the first database snapshot page, the source database node performs the write operation in the first copy-on-write file; The destination database node uses the database snapshot page to execute the received transaction based on a copy-on-write mode, including: If the write operation of the transaction to be executed by the destination database node is the first write operation to the second database file, creating a corresponding second copy-on-write file for the second database file, where the second database file is the database file of the destination database node; If the write operation of the transaction to be executed by the destination database node is the first write operation on the second database snapshot page, create a corresponding second copy-on-write file page in the second copy-on-write file, copy data of the second database snapshot page to the second copy-on-write file page, where the second database snapshot page is one or more of the database snapshot pages, and the destination database node performs the write operation in the second copy-on-write file; If the write operation of the transaction to be executed by the destination database node is not the first write operation on the second database snapshot page, the destination database node performs the write operation in the second copy-on-write file; The method further comprises: After the source database node creates the first copy-on-write file page, writing the first database snapshot page corresponding to the first copy-on-write file page back to the shared storage area, and writing the first copy-on-write file page into a local cache pool; After the destination database node creates the second copy-on-write file page, the second database snapshot page corresponding to the second copy-on-write file page is deleted, and the second copy-on-write file page is written to the local cache pool.

6. The method according to claim 1, further comprising: The source database node modifies the transaction state information in the third copy-on-write file by copying relevant data pages in the transaction state maintenance file based on the copy-on-write mode, wherein the transaction state maintenance file is used to maintain the transaction state information and is stored in the shared storage area; The destination database node modifies the transaction status information in the fourth copy-on-write file by copying the relevant data pages in the transaction status maintenance file based on the copy-on-write mode.

7. The method according to claim 1, further comprising: After the snapshot operation starts, the source database node sends the next available transaction unique identifier and transaction global variable information to the destination database node, so that the destination database node maintains the available transaction unique identifier and transaction global variable information; When the source database node needs an available transaction unique identifier, the destination database node allocates the available transaction unique identifier to the source database node; The routing of the transaction received after the snapshot operation to the destination database node includes: The proxy server suspends the transaction received during the execution of the snapshot operation on the source database node; When the source database node finishes executing the snapshot operation, the proxy server assigns a start timestamp to the suspended transaction; The proxy server routes the transaction assigned with the start timestamp to the destination database node.

8. The method according to claim 1, further comprising: When the destination database node reclaims memory space based on the multi-version transaction concurrency control mechanism, if the database snapshot page of the version to be reclaimed is currently being used by the transaction of the source database node, the destination database node abandons the reclamation of the database snapshot page of the version to be reclaimed.

9. A database migration method, applied to a source database node, comprising: Suspend the executing transaction, perform snapshot operation, and obtain the database snapshot page; After completing the snapshot operation, the database snapshot page is used to resume executing transactions based on the copy-on-write mode, and the snapshot operation is used to route transactions received after the snapshot operation to the destination database node; The database snapshot page is further configured to enable a destination database node to execute the received transaction using the database snapshot page based on a copy-on-write mode after receiving the transaction, wherein the destination database node and the source database node are database instances based on a shared storage area for storing database data; Sending the modified content of the write operation performed after the snapshot operation to the destination database node, so that the destination database node performs logical playback using the modified content; When determining that the logic playback of the destination database node is successfully completed, committing the transaction corresponding to the modified content; The destination database node performs logical playback using the modified content, including: The destination database node initiates a corresponding shadow transaction for the transaction to which the modified content belongs, wherein the shadow transaction includes: a read operation on a database snapshot page related to the modified content; write conflict detection, lock conflict detection, and constraint condition check on data read by the read operation; and, if the write conflict detection, lock conflict detection, and constraint condition check all pass, a logical playback operation using the modified content. The read operation uses the same snapshot timestamp as that of the transaction to which the modified content belongs to read the database snapshot page; If any one of the write conflict detection, lock conflict detection and constraint condition check fails, the destination database node terminates execution of the shadow transaction; If the shadow transaction successfully completes the logic replay operation, the destination database node assigns a commit timestamp to the shadow transaction, commits the shadow transaction, and sends the commit timestamp to the source database node, so that the source database node commits the corresponding transaction using the commit timestamp; If the logical playback of the shadow transaction fails or is aborted, the destination database node returns failure information to the source database node, causing the source database node to roll back the corresponding transaction.

10. A database migration method, applied to a destination database node, comprising: receiving a transaction, wherein the transaction is a transaction that is routed to the destination database node after the source database node performs a snapshot operation and obtains a database snapshot page, and the destination database node and the source database node are database instances based on a shared storage area for storing database data; Executing the received transaction based on a copy-on-write mode using the database snapshot page, wherein the database snapshot page is further used to enable the source database node to resume executing the transaction based on the copy-on-write mode using the database snapshot page; Obtaining modification content corresponding to the write operation performed by the source database node after the snapshot operation; Performing logic playback using the modified content, so that the source database node commits the transaction corresponding to the modified content when determining that the logic playback of the destination database node is successfully completed; The destination database node performs logical playback using the modified content, including: The destination database node initiates a corresponding shadow transaction for the transaction to which the modified content belongs, wherein the shadow transaction includes: a read operation on a database snapshot page related to the modified content; write conflict detection, lock conflict detection, and constraint condition check on data read by the read operation; and, if the write conflict detection, lock conflict detection, and constraint condition check all pass, a logical playback operation using the modified content. The read operation uses the same snapshot timestamp as that of the transaction to which the modified content belongs to read the database snapshot page; If any one of the write conflict detection, lock conflict detection and constraint condition check fails, the destination database node terminates execution of the shadow transaction; If the shadow transaction successfully completes the logic replay operation, the destination database node assigns a commit timestamp to the shadow transaction, commits the shadow transaction, and sends the commit timestamp to the source database node, so that the source database node commits the corresponding transaction using the commit timestamp; If the logical playback of the shadow transaction fails or is aborted, the destination database node returns failure information to the source database node, causing the source database node to roll back the corresponding transaction.

11. A database migration system, comprising: A source database node implemented by the database migration method according to any one of claims 1 to 10, and a destination database node implemented by the database migration method according to any one of claims 1 to 10.

12. A computing device comprising: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the source database node and / or the destination database node in the database migration system according to claim 11 are implemented.

13. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the database migration method according to any one of claims 1 to 10.

Images