Data migration method and device, electronic equipment, storage medium and product

Abstract

The invention discloses a data migration method and device, electronic equipment, a storage medium and a product. The method comprises the steps of determining migration metadata and the number of batch migration leaf nodes in response to triggering of a data migration event; wherein the migration metadata comprises to-be-migrated partition data, a source node currently storing the to-be-migrated partition data, a target node to which the to-be-migrated data is planned to be migrated and a migration mode; based on the number of batch migration leaf nodes, determining target partition data of each batch migration from the to-be-migrated partition data; and according to the sequence of batches from front to back, migrating the target partition data migrated in each batch from the source node to the target node in sequence based on a migration mode, and constructing a non-leaf node corresponding to the leaf node where the target partition data migrated in the current batch is located in the B + tree in the target node based on a B + tree construction strategy. According to the scheme, partition data can be migrated in batches, and load balancing of nodes in a distributed database system is achieved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data migration method, device, electronic device, storage medium and product. Background Art

[0002] In a distributed database system, data is typically divided into multiple partitions and stored in a computer node in a computer cluster. Each computer node in the computer cluster can store one or more partitions to achieve horizontal scaling and load balancing of the computer nodes. The B+ tree is a widely used index structure used to accelerate data search and sorting. However, as data grows and access patterns change, the load on different computer nodes in a computer cluster may become unbalanced, causing some computer nodes to be overloaded while others are lightly loaded, which directly affects the overall performance and resource utilization of the distributed database system. Therefore, in a distributed database system, a key step in achieving load balancing is to be able to efficiently migrate certain partitioned data in a computer node that uses a B+ tree as an index structure to another computer node, and to do so without affecting data operations. Summary of the Invention

[0003] The present invention provides a data migration method, device, electronic device, storage medium and product, which can migrate partitioned data in batches and achieve load balancing of nodes in a distributed database system.

[0004] According to one aspect of the present invention, a data migration method is provided, comprising:

[0005] In response to a data migration event being triggered, determining migration metadata and the number of leaf nodes to be migrated in a batch; wherein the migration metadata includes partition data to be migrated, a source node currently storing the partition data to be migrated, a target node to which the data to be migrated is planned to be migrated, and a migration method; the partition data to be migrated is stored in the source node in a B+ tree format;

[0006] Determining target partition data for each batch migration from the partition data to be migrated based on the number of leaf nodes in the batch migration;

[0007] In the order of batches from front to back, the target partition data of each batch migration is migrated from the source node to the target node in sequence based on the migration method, and a non-leaf node corresponding to the leaf node in the B+ tree where the target partition data of the current batch migration is located is constructed in the target node based on the B+ tree construction strategy.

[0008] According to another aspect of the present invention, there is provided a data migration device, comprising:

[0009] A migration metadata determination module is configured to determine migration metadata and the number of leaf nodes to be migrated in a batch in response to a data migration event being triggered; wherein the migration metadata includes the partition data to be migrated, the source node currently storing the partition data to be migrated, the target node to which the data to be migrated is planned to be migrated, and the migration method; the partition data to be migrated is stored in the source node in the form of a B+ tree;

[0010] a target partition data determination module, configured to determine target partition data for each batch of migration from the partition data to be migrated based on the number of leaf nodes in the batch migration;

[0011] The target partition data migration module is used to migrate the target partition data of each batch from the source node to the target node in sequence according to the migration method in the order of batches from front to back, and to construct a non-leaf node in the target node corresponding to the leaf node in the B+ tree where the target partition data of the current batch migration is located based on the B+ tree construction strategy.

[0012] According to another aspect of the present invention, an electronic device is provided, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the data migration method described in any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data migration method described in any embodiment of the present invention when executed.

[0017] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the data migration method according to any embodiment of the present invention is implemented.

[0018] The data migration scheme of the embodiment of the present invention determines the migration metadata and the number of leaf nodes for batch migration in response to a data migration event being triggered; wherein the migration metadata includes the partition data to be migrated, the source node currently storing the partition data to be migrated, the target node to which the data to be migrated is planned to be migrated, and the migration method; the partition data to be migrated is stored in the source node in the form of a B+ tree; based on the number of leaf nodes for batch migration, the target partition data for each batch migration is determined from the partition data to be migrated; in order from front to back of the batch, the target partition data of each batch migration is migrated from the source node to the target node based on the migration method, and a non-leaf node corresponding to the leaf node in the B+ tree where the target partition data of the current batch migration is located is constructed in the target node based on the B+ tree construction strategy. Through the technical solution provided by the embodiment of the present invention, partition data can be migrated in batches to achieve load balancing of nodes in a distributed database system, and at the same time, the impact on data operations can be minimized when migrating partition data.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flowchart of a data migration method provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of the structure of a data migration device provided by an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of the structure of an electronic device for implementing the data migration method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] Figure 1 A flowchart of a data migration method is provided for an embodiment of the present invention. This embodiment is applicable to the case of migrating partition data. The method can be executed by a data migration device. The data migration device can be implemented in the form of hardware and / or software. The data migration device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0027] S110. In response to a data migration event being triggered, determine the migration metadata and the number of leaf nodes to be migrated in a batch; wherein, the migration metadata includes the partition data to be migrated, the source node currently storing the partition data to be migrated, the target node to which the data to be migrated is planned to be migrated, and the migration method; the partition data to be migrated is stored in the source node in the form of a B+ tree.

[0028] In an embodiment of the present invention, when a data migration request is detected, it is determined that a data migration event is triggered. Optionally, when it is detected that the data distribution in the distributed database system is uneven, it is determined that a data migration event is triggered. In response to the data migration event being triggered, migration metadata is determined, wherein the migration metadata is data information associated with the partition data migration, and the migration metadata includes the partition data to be migrated, the source node currently storing the partition data to be migrated, the target node to which the data to be migrated is planned to be migrated, and the migration method. Among them, the partition data to be migrated can be represented by the partition data ID, the computer device node currently storing the partition data to be migrated can be represented by the source node ID, and the computer device node to which the partition data to be migrated is planned to be migrated can be represented by the target node ID. It should be noted that since the source node may contain multiple B+ trees, the partition data to be migrated can be the partition data stored in all or part of the leaf nodes in a B+ tree in the source node, or it can be the partition data stored in all or part of the leaf nodes in multiple B+ trees. The migration method can include a multi-version concurrency control mechanism (MVCC) method and a non-MVCC method. It should be noted that the non-MVCC method is applicable to databases that do not support MVCC and databases that support MVCC, and the MVCC method is only applicable to databases that support MVCC. In an embodiment of the present invention, in response to a data migration event being triggered, in addition to determining the migration metadata, the number of leaf nodes for batch migration is also determined, where the number of leaf nodes for batch migration can be understood as the number of leaf nodes in the B+ tree for each batch migration. The number of leaf nodes for batch migration can be a parameter pre-set by the user.

[0029] S120 : Determine target partition data for each batch migration from the partition data to be migrated based on the number of leaf nodes in the batch migration.

[0030] In an embodiment of the present invention, since the partition data to be migrated is stored in the source node in the form of a B+ tree, and each B+ tree contains multiple leaf nodes, the target partition data for each batch migration can be determined from the partition data to be migrated based on the number of leaf nodes migrated in the batch. Exemplarily, the partition data to be migrated is the partition data corresponding to all leaf nodes stored in a B+ tree in the source node, and the B+ tree contains 10 leaf nodes. If the number of leaf nodes migrated in a batch is 2, the data to be migrated corresponding to the 10 leaf nodes in the B+ tree are divided into target partition data for migration in 5 batches according to the order of the leaf nodes from left to right. Among them, according to the order of the leaf nodes from left to right, the data to be migrated stored in the 1st leaf node and the 2nd leaf node in the B+ tree are the target partition data for the first batch of migration; the data to be migrated stored in the 3rd leaf node and the 4th leaf node in the B+ tree are the target partition data for the second batch of migration; the data to be migrated stored in the 5th leaf node and the 6th leaf node in the B+ tree are the target partition data for the third batch of migration; the data to be migrated stored in the 7th leaf node and the 8th leaf node in the B+ tree are the target partition data for the fourth batch of migration; and the data to be migrated stored in the 9th leaf node and the 10th leaf node in the B+ tree are the target partition data for the fifth batch of migration. As another example, the partition data to be migrated is the partition data corresponding to all leaf nodes stored in the first B+ tree in the source node and the partition data corresponding to the first three leaf nodes stored in the second B+ tree from left to right. The first B+ tree contains 6 leaf nodes. If the number of leaf nodes to be migrated in a batch is 3, then the data to be migrated corresponding to the 6 leaf nodes in the first B+ tree are divided into target partition data for migration in two batches according to the order of the leaf nodes from left to right, and the data to be migrated corresponding to the first three leaf nodes in the second B+ tree are divided into target partition data for migration in one batch. Among them, according to the order of the leaf nodes from left to right, the data to be migrated stored in the 1st to 3rd leaf nodes in the first B+ tree can be the target partition data for the first batch of migration; the data to be migrated stored in the 4th to 6th leaf nodes in the first B+ tree can be the target partition data for the second batch of migration; and the data to be migrated stored in the 1st to 3rd leaf nodes in the second B+ tree can be the target partition data for migration in the third batch. Optionally, the data to be migrated stored in the 1st leaf node to the 3rd leaf node in the second B+ tree can be used as the target partition data for the first batch of migration in the order of leaf nodes from left to right; the data to be migrated stored in the 1st leaf node to the 3rd leaf node in the first B+ tree can be used as the target partition data for the second batch of migration; and the data to be migrated stored in the 4th leaf node to the 6th leaf node in the first B+ tree can be used as the target partition data for the third batch of migration.It should be noted that if the to-be-migrated partition data is the partition data stored in the leaf nodes of the plurality of B+ trees in the source node, the order of the migration batches between the target partition data of each batch corresponding to the plurality of B+ trees is not limited.

[0031] S130, in the order from the front to the back of the batches, the target partition data migrated by each batch is sequentially migrated from the source node to the target node based on the migration mode, and the non-leaf nodes corresponding to the leaf nodes in the B+ tree in which the target partition data of the current batch is migrated are constructed in the target node based on the B+ tree construction strategy.

[0032] In the embodiment of the application, for the target partition data migrated by each batch in the order from the front to the back of the batches, the target partition data migrated by the current batch is sequentially migrated from the source node to the target node based on the migration mode, and after the target partition data migrated by the current batch is migrated to the target node, the non-leaf nodes corresponding to the leaf nodes in the B+ tree in which the target partition data of the current batch is migrated are constructed in the target node based on the B+ tree construction strategy. Since the number of non-leaf nodes in the B+ tree is relatively small, for example, there is only one non-leaf node in the B+ tree with a height of 2, after the target partition data migrated by the current batch is migrated to the target node, the non-leaf nodes corresponding to the leaf nodes in the B+ tree in which the target partition data of the current batch is migrated are reconstructed in the target node based on the B+ tree construction strategy, rather than directly migrating the non-leaf nodes, not only the cost is lower, but also the problem of address information mismatch and complex address mapping caused by direct migration can be effectively avoided. In addition, after the target partition data migrated by the current batch is migrated to the target node, the non-leaf nodes corresponding to the leaf nodes in the B+ tree in which the target partition data of the current batch is migrated are directly reconstructed in the target node, without waiting for all the target partition data migrated by other batches to be completely migrated before reconstructing the non-leaf nodes, the target partition data that has been migrated to the target node can be processed, not only simplifying the data query process, but also enhancing the query efficiency and system stability. It can be understood that the to-be-migrated partition data is migrated in batches, that is, after the target partition data of the first batch is migrated from the source node to the target node based on the migration mode, the target partition data of the second batch is migrated from the source node to the target node, and so on, until the migration operation of all to-be-migrated partition data is completed.

[0033] Optionally, the target partition data of each batch migration is migrated from the source node to the target node in sequence according to the migration method in the order from the front to the back of the batch, including: for the target partition data of each batch migration in the order from the front to the back of the batch, if the migration method is the multi-version concurrency control mechanism MVCC method, a page snapshot corresponding to the leaf node of the target partition data of the current batch migration in the B+ tree is created in the source node, and the target partition data of the current batch migration in the page snapshot is sent to the target node; the source node obtains the transaction log generated after the page snapshot is created, and sends the transaction log to the target node; the target node adjusts the target partition data of the current batch migration based on the transaction log.

[0034] Exemplarily, for the target partition data of each batch migration in the order from the front to the back of the batch, if the migration method is MVCC, a page snapshot corresponding to the leaf node in the B+ tree of the target partition data of the current batch migration is created in the source node, wherein the page snapshot contains the state of the target partition data of the current batch migration at the time point when the page snapshot is created. Due to the MVCC mechanism, the created page snapshot can be isolated from the currently running transaction and will not block the ongoing read and write operations on the target partition data of the current batch migration in the source node. The target partition data of the current batch migration in the page snapshot is sent to the target node, wherein the target partition data of the current batch migration in the page snapshot can be converted into a text format (such as CSV format or JSON format), and then the target partition data of the current batch migration in text format is exported from the source node and transmitted to the target node. Optionally, the target partition data of the current batch migration in the page snapshot can also be directly transmitted to the target node in binary format. The advantage of such a setting is that it greatly improves the data transmission speed and efficiency and reduces the time required for data migration. Starting from the time when the page snapshot corresponding to the leaf node in the B+ tree of the target partition data of the current batch migration is created on the source node, all transaction logs (Binlog) are captured and sent to the target node. Since the transaction log contains all modification information of the target partition data of the current batch migration after the page snapshot is created, the target node adjusts the target partition data of the current batch migration obtained based on the page snapshot based on the transaction log to ensure that the target node can receive all updates to the target partition data of the current batch migration after the page snapshot is created, thereby ensuring the consistency of the target partition data of the current batch migration on the source node and the target node before and after the migration. After confirming the consistency of the target partition data of the current batch migration before and after the migration, the target node is set as the master node to take on subsequent read and write requests for the target partition data of the current batch migration. At the same time, the source node can continue to serve as a slave node until it is completely stopped or the next batch of migration operations is carried out.

[0035] In an embodiment of the present invention, during the migration of data to be migrated based on the MVCC method, the write operation of the partitioned data in the source node is not blocked at all. In addition, since the data to be migrated is migrated in batches, the disadvantages of the distributed database system having to retain all historical versions with version numbers greater than or equal to the page snapshot version number for a long time and the long-term accumulation of transaction logs are overcome.

[0036] Optionally, the target partition data of each batch migration is migrated from the source node to the target node in sequence according to the migration method in the order from the front to the back of the batch, including: for the target partition data of each batch migration in the order from the front to the back of the batch, if the migration method is non-MVCC, then after controlling the target partition data of the current batch migration in the source node to be in a read-only state, the source node sends the target partition data of the current batch migration to the target node; the target node receives and saves the target partition data of the current batch migration.

[0037] Exemplarily, for the target partition data of each batch migration in the order from the front to the back of the batch, if the migration method is non-MVCC, the target partition data of the current batch migration in the source node is controlled to be in a read-only state. Exemplarily, a read-only lock is applied to the target partition data of the current batch migration, and new write operations are blocked until all current non-read operations are completed, thereby achieving successful application of the read-only lock, that is, successfully adjusting the target partition data of the current batch migration in the source node to a read-only state. After adjusting the target partition data of the current batch migration in the source node to a read-only state, the source node sends the target partition data of the current batch migration to the target node, and the target node receives and saves the target partition data of the current batch migration, thereby completing the migration operation of the target partition data of the current batch migration. Among them, the target partition data of the current batch migration can be converted into a text format (such as CSV format or JSON format), and then the target partition data of the current batch migration in text format is exported from the source node and transmitted to the target node. Optionally, the target partition data of the current batch migration can also be directly transmitted to the target node in binary format. Optionally, after migrating the target partition data of the current batch migration from the source node to the target node, the target partition data of the current batch migration in the source node is marked as unreadable, and the target partition data of the current batch migration in the target node is marked as readable and writable, which officially takes effect and becomes the new data source, so that any subsequent read requests for the target partition data of the current batch migration will be redirected to the target node.

[0038] In an embodiment of the present invention, the data to be migrated is migrated in batches based on a non-MVCC method. Since only the target partition data of the current batch migration needs to be read, there is no need to transmit the transaction log after the page snapshot is created, and the target node does not need to update the target data of the current batch migration obtained based on the page snapshot based on the transaction log, thereby realizing rapid migration of partition data. In addition, although migrating partition data based on a non-MVCC method will prevent write operations on the partition data of the batch being migrated, since the partition data to be migrated is migrated in batches, the range of partition data that cannot be written is very small and the time is also very short. In an embodiment of the present invention, the solution of migrating partition data in batches based on a non-MVCC method can be applied to scenarios where partition data writing is infrequent during the migration process, or where a short pause in partition data writing is acceptable.

[0039] The data migration method of the embodiment of the present invention determines the migration metadata and the number of leaf nodes of the batch migration in response to the triggering of the data migration event; wherein, the migration metadata includes the partition data to be migrated, the source node currently storing the partition data to be migrated, the target node to which the data to be migrated is planned to be migrated, and the migration method; the partition data to be migrated is stored in the source node in the form of a B+ tree; based on the number of leaf nodes of the batch migration, the target partition data of each batch migration is determined from the partition data to be migrated; in order from front to back of the batch, the target partition data of each batch migration is migrated from the source node to the target node based on the migration method, and the non-leaf node corresponding to the leaf node of the target partition data of the current batch migration in the B+ tree is constructed in the target node based on the B+ tree construction strategy. Through the technical solution provided by the embodiment of the present invention, partition data can be migrated in batches to achieve load balancing of nodes in a distributed database system, and at the same time, the impact on data operations can be minimized when migrating partition data.

[0040] In some embodiments, in the process of migrating the target partition data of each batch migration from the source node to the target node in sequence based on the migration method, it also includes: determining in real time the first key value of the partition data that has been migrated and the second key value of the partition data being migrated; when receiving a partition data processing request for any partition data in the partition data to be migrated, determining the partition data to be processed indicated by the partition data processing request and determining the third key value of the partition data to be processed; and performing processing operations on the partition data to be processed based on the first key value, the second key value, and the third key value. The advantage of this setting is that it ensures the feasibility of online partition data migration and minimizes the impact of inoperability of the partition data to be migrated.

[0041] In an embodiment of the present invention, the first key value of the partitioned data that has been migrated and the second key value of the partitioned data being migrated are determined in real time, wherein the first key value is used to characterize the progress of the partitioned data migration, the first key value is the key value boundary value of a leaf node of the B+ tree in the source node, and the second key value is the boundary value of the key value range of the partitioned data currently being migrated, and the second key value is the key value boundary value of a leaf node of the B+ tree in the source node. It can be understood that as the partitioned data is continuously migrated, the first key value and the second key value are dynamically updated. In an embodiment of the present invention, when a partitioned data processing request for any partitioned data in the partitioned data to be migrated is received, the partitioned data to be processed indicated by the partitioned data processing request is determined. For example, the partitioned data to be processed can be determined based on the ID of the partitioned data included in the partitioned data processing request, and the third key value of the partitioned data to be processed is determined, wherein the third key value is the key value of the partitioned data to be processed in the leaf node of the B+ tree where it is located. It should be noted that the processing operation corresponding to the partitioned data processing request for the partitioned data to be processed may include any one of read, modify, add and delete operations. The third key value is compared with the first key value and the second key value respectively, and a processing operation is performed on the partition data to be processed according to the comparison results.

[0042] Optionally, a processing operation is performed on the partition data to be processed based on the first key value, the second key value, and the third key value, including: if the third key value is less than the first key value, performing a processing operation on the partition data to be processed in the target node that matches the partition data processing request; if the third key value is greater than or equal to the first key value and less than or equal to the second key value, performing a processing operation on the partition data to be processed based on the migration method; if the third key value is greater than the second key value, performing a processing operation on the partition data to be processed in the source node that matches the partition data processing request. The advantage of this setting is that it can accurately control which data can be read and written and which data is limited to reading based on the key values ​​of the partition data that has been migrated and the key values ​​of the partition data being migrated, thereby achieving seamless coordination between data migration and data processing operations.

[0043] Optionally, if the third key value is greater than or equal to the first key value and less than or equal to the second key value, a processing operation is performed on the partition data to be processed based on the migration method, including: if the third key value is greater than or equal to the first key value and less than or equal to the second key value, then when the migration method is the MVCC method, a processing operation matching the partition data processing request is performed on the partition data to be processed in the source node; if the third key value is greater than or equal to the first key value and less than or equal to the second key value, then when the migration method is the non-MVCC method, if the processing operation matching the partition data processing request is a read operation, then a read operation is performed on the partition data to be processed in the source node; if the processing operation matching the partition data processing request is an operation other than a read operation, then processing operations on the partition data to be processed are prohibited.

[0044] Exemplarily, if the third key value is less than the first key value, it means that the partition data to be processed has been migrated from the source data to the target node. Therefore, a processing operation matching the partition data processing request is performed on the partition data to be processed in the target node; wherein the processing operation may include any one of reading, modifying, adding, and deleting. If the third key value is greater than the second key value, it means that a migration operation has not been performed on the partition data to be processed. Therefore, a processing operation matching the partition data processing request is performed on the partition data to be processed in the source node; wherein the processing operation may include any one of reading, modifying, adding, and deleting. If the third key value is greater than or equal to the first key value and less than or equal to the second key value, it means that a migration operation is currently being performed on the partition data to be processed. Therefore, a processing operation is performed on the partition data to be processed based on the migration method.

[0045] Exemplarily, when the migration mode is MVCC mode, a processing operation matching the partition data processing request is performed on the partition data to be processed in the source node; wherein the processing operation may include any one of reading, modifying, adding and deleting. It can be understood that if the third key value is greater than or equal to the first key value and less than or equal to the second key value, when the migration mode is MVCC mode, regardless of whether the processing operation on the partition data to be processed is a read operation or any one of the operations such as modification, adding and deleting, the corresponding processing operation can be directly performed on the partition data to be processed in the source node. If the third key value is greater than or equal to the first key value and less than or equal to the second key value, then when the migration mode is non-MVCC mode, if the processing operation matching the partition data processing request is a read operation, then a read operation is performed on the partition data to be processed in the source node; if the processing operation matching the partition data processing request is an operation other than a read operation, such as a modification, adding and deleting operation, then the processing operation on the partition data to be processed is prohibited. It can be understood that if the third key value is greater than or equal to the first key value and less than or equal to the second key value, when the migration method is MVCC, only read operations can be performed on the partition data to be processed in the source node, and operations such as modification, addition and deletion cannot be performed.

[0046] Figure 2 A schematic diagram of the structure of a data migration device provided by an embodiment of the present invention. Figure 2 As shown, the device includes:

[0047] The migration metadata determination module 210 is configured to determine the migration metadata and the number of leaf nodes to be migrated in a batch in response to a data migration event being triggered. The migration metadata includes the partition data to be migrated, the source node currently storing the partition data to be migrated, the target node to which the partition data to be migrated is planned to be migrated, and the migration method. The partition data to be migrated is stored in the source node in the form of a B+ tree.

[0048] A target partition data determination module 220 is configured to determine target partition data for each batch of migration from the partition data to be migrated based on the number of leaf nodes in the batch migration;

[0049] The target partition data migration module 230 is used to migrate the target partition data of each batch migration from the source node to the target node in sequence according to the order of batches from front to back based on the migration method, and to construct a non-leaf node in the target node corresponding to the leaf node in the B+ tree where the target partition data of the current batch migration is located based on the B+ tree construction strategy.

[0050] Optional target partition data migration module, used to:

[0051] For the target partition data migrated in each batch in order from the front to the back, if the migration mode is the multi-version concurrency control mechanism MVCC mode, create a page snapshot corresponding to the leaf node of the target partition data of the current batch migration in the B+ tree in the source node, and send the target partition data of the current batch migration in the page snapshot to the target node;

[0052] The source node obtains a transaction log generated after the page snapshot is created, and sends the transaction log to the target node;

[0053] The target node adjusts the target partition data of the current batch migration based on the transaction log.

[0054] Optional target partition data migration module, used to:

[0055] For the target partition data of each batch to be migrated in order from the front to the back of the batch, if the migration mode is non-MVCC, after controlling the target partition data of the current batch to be migrated in the source node to be in a read-only state, the source node sends the target partition data of the current batch to the target node;

[0056] The target node receives and stores the target partition data of the current batch migration.

[0057] Optionally, in the process of sequentially migrating the target partition data of each batch from the source node to the target node based on the migration method, the process further includes:

[0058] A first key value and a second key value determination module, configured to determine in real time the first key value of the partition data that has been migrated and the second key value of the partition data that is being migrated;

[0059] a third key value determining module configured to, upon receiving a partition data processing request for any partition data among the partition data to be migrated, determine the partition data to be processed indicated by the partition data processing request and determine a third key value of the partition data to be processed;

[0060] The partition data processing module to be processed is used to perform a processing operation on the partition data to be processed based on the first key value, the second key value and the third key value.

[0061] Optionally, the partition data processing module to be processed includes:

[0062] a first processing unit, configured to perform, in the target node, a processing operation matching the partition data processing request on the partition data to be processed if the third key value is less than the first key value;

[0063] a second processing unit, configured to perform a processing operation on the to-be-processed partition data based on the migration mode if the third key value is greater than or equal to the first key value and less than or equal to the second key value.

[0064] a third processing unit, configured to perform a processing operation on the to-be-processed partition data in the source node that matches the partition data processing request if the third key value is greater than the second key value.

[0065] Optionally, the second processing unit is configured to:

[0066] if the third key value is greater than or equal to the first key value and less than or equal to the second key value, when the migration mode is an MVCC mode, perform a processing operation on the to-be-processed partition data in the source node that matches the partition data processing request.

[0067] if the third key value is greater than or equal to the first key value and less than or equal to the second key value, when the migration mode is a non-MVCC mode, if the processing operation that matches the partition data processing request is a read operation, perform a read operation on the to-be-processed partition data in the source node, and if the processing operation that matches the partition data processing request is an operation other than a read operation, prohibit performing a processing operation on the to-be-processed partition data.

[0068] The data migration apparatus provided by the embodiments of the present application can perform the data migration method provided by any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of performing the method.

[0069] Figure 3 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0070] As Figure 3As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0071] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0072] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the data migration method.

[0073] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication unit 19, or installed from the storage unit 18, or installed from the ROM 12. When the computer program is executed by the processor 11, the above-mentioned functions defined in the method of the embodiment of the present invention are performed.

[0074] In some embodiments, the data migration method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data migration method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the data migration method in any other suitable manner (e.g., via firmware).

[0075] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0076] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0077] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0078] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0079] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0080] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0081] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0082] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data migration method, characterized in that: include: In response to a data migration event being triggered, determining migration metadata and the number of leaf nodes to be migrated in a batch; wherein the migration metadata includes partition data to be migrated, a source node currently storing the partition data to be migrated, a target node to which the data to be migrated is planned to be migrated, and a migration method; the partition data to be migrated is stored in the source node in a B+ tree format; Determining target partition data for each batch migration from the partition data to be migrated based on the number of leaf nodes in the batch migration; In the order of batches from front to back, the target partition data of each batch migration is migrated from the source node to the target node in sequence based on the migration method, and a non-leaf node corresponding to the leaf node in the B+ tree where the target partition data of the current batch migration is located is constructed in the target node based on the B+ tree construction strategy.

2. The method according to claim 1, characterized in that Migrating the target partition data of each batch from the source node to the target node in sequence based on the migration method in a batch-by-batch order, including: For the target partition data migrated in each batch in order from the front to the back, if the migration mode is the multi-version concurrency control mechanism MVCC mode, create a page snapshot corresponding to the leaf node of the target partition data of the current batch migration in the B+ tree in the source node, and send the target partition data of the current batch migration in the page snapshot to the target node; The source node obtains a transaction log generated after the page snapshot is created, and sends the transaction log to the target node; The target node adjusts the target partition data of the current batch migration based on the transaction log.

3. The method according to claim 1, characterized in that Migrating the target partition data of each batch from the source node to the target node in sequence based on the migration method in a batch-by-batch order, including: For the target partition data of each batch to be migrated in order from the front to the back of the batch, if the migration mode is non-MVCC, after controlling the target partition data of the current batch to be migrated in the source node to be in a read-only state, the source node sends the target partition data of the current batch to the target node; The target node receives and stores the target partition data of the current batch migration.

4. The method according to claim 1, wherein In the process of sequentially migrating the target partition data of each batch from the source node to the target node based on the migration method, the process further includes: Determine in real time the first key value of the partition data that has been migrated and the second key value of the partition data that is being migrated; When a partition data processing request for any partition data among the partition data to be migrated is received, determining the partition data to be processed indicated by the partition data processing request, and determining a third key value of the partition data to be processed; A processing operation is performed on the partition data to be processed based on the first key value, the second key value, and the third key value.

5. The method according to claim 4, characterized in that Performing a processing operation on the partition data to be processed based on the first key value, the second key value, and the third key value includes: If the third key value is less than the first key value, performing a processing operation matching the partition data processing request on the to-be-processed partition data in the target node; If the third key value is greater than or equal to the first key value and less than or equal to the second key value, processing the partition data to be processed based on the migration method; If the third key value is greater than the second key value, a processing operation matching the partition data processing request is performed on the partition data to be processed in the source node.

6. The method according to claim 5, characterized in that If the third key value is greater than or equal to the first key value and less than or equal to the second key value, processing the partition data to be processed based on the migration method includes: If the third key value is greater than or equal to the first key value and less than or equal to the second key value, when the migration mode is MVCC mode, performing a processing operation matching the partition data processing request on the to-be-processed partition data in the source node; If the third key value is greater than or equal to the first key value and less than or equal to the second key value, when the migration mode is non-MVCC mode, if the processing operation that matches the partition data processing request is a read operation, then the read operation is performed on the partition data to be processed in the source node; if the processing operation that matches the partition data processing request is any operation other than a read operation, then the processing operation on the partition data to be processed is prohibited.

7. A data migration device, characterized in that: include: A migration metadata determination module is configured to determine migration metadata and the number of leaf nodes to be migrated in a batch in response to a data migration event being triggered; wherein the migration metadata includes the partition data to be migrated, the source node currently storing the partition data to be migrated, the target node to which the data to be migrated is planned to be migrated, and the migration method; the partition data to be migrated is stored in the source node in the form of a B+ tree; a target partition data determination module, configured to determine target partition data for each batch of migration from the partition data to be migrated based on the number of leaf nodes in the batch migration; The target partition data migration module is used to migrate the target partition data of each batch from the source node to the target node in sequence according to the migration method in the order of batches from front to back, and to construct a non-leaf node in the target node corresponding to the leaf node in the B+ tree where the target partition data of the current batch migration is located based on the B+ tree construction strategy.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the data migration method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data migration method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the data migration method according to any one of claims 1 to 6.

Images