A data partition adjustment method and system

Abstract

The application discloses a data partition adjustment method and system, the method comprising: copying uncommitted access list and concurrency control logic of a to-be-migrated partition basket from an original node to a destination node; the original node is a node where an original data partition is located; the destination node is a node where a destination data partition is located; creating a shadow of the to-be-migrated partition basket in the destination data partition; the shadow comprises a list of data in different storage states; routing an access request for the data basket to the destination node, so that an operation request for the to-be-migrated data basket during running is directed to the shadow; the destination node processes the access request for the partition basket based on the uncommitted access list and the concurrency control logic; and migrating data of the to-be-migrated partition basket to the destination data partition based on the list of data in different storage states.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data partitioning adjustment method and system. Background Technology

[0002] With the development of information technology, more and more service platforms are using clusters to process massive amounts of data for business services. The computing and storage resources of a single server are limited, and data partitioning algorithms can distribute data across different servers for computation and storage. This can significantly reduce the demands on the computing and storage resources of a single server. However, in data partitioning scenarios, uneven initial partitioning can lead to uneven demands on computing and storage resources. Therefore, after a period of operation, data partitioning often needs to be readjusted.

[0003] Currently, when data partitions are readjusted, access requests need to be stopped. This prevents data partition readjustment from being performed during peak business periods. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a data partitioning adjustment method and system that can dynamically adjust data partitions at runtime, thereby improving the overall resource utilization of the entire cluster.

[0005] This application discloses the following technical solution:

[0006] The first aspect of this application provides a data partitioning adjustment method, including:

[0007] Copy the uncommitted access list and concurrency control logic of the partition basket to be migrated from the original node to the destination node; the original node is the node where the original data partition is located; the destination node is the node where the destination data partition is located.

[0008] Create a shadow of the basket of partitions to be migrated in the destination data partition; the shadow includes a list of data in different storage states;

[0009] Access requests to the data basket are routed to the destination node so that operation requests for the data basket to be migrated during the operation are directed to the shadow.

[0010] The destination node processes access requests for the partition basket based on the uncommitted access list and concurrency control logic.

[0011] Based on the list of data in different storage states, the data in the partition basket to be migrated will be migrated to the destination data partition.

[0012] In one possible implementation, routing the access request to the data basket to the destination node includes:

[0013] In the data partition location determination terminal, the node where the basket of the partition to be migrated is located is modified to the destination node; the data partition location determination terminal includes a client and / or a proxy server.

[0014] In one possible implementation, the list of data in different storage states includes: a first dirty page list recording dirty page data of the partition basket stored in the memory block of the origin node but not in the persistent storage block; a second dirty page list recording dirty page data of the partition basket stored in the memory block of the destination node but not in the persistent storage block; a first memory block list recording the memory block of the partition basket that has been copied in the origin node; and a second memory block list recording the memory block of the partition basket that is in the destination node.

[0015] The list of data based on the different storage states migrates the data from the partition basket to the destination data partition, including:

[0016] The destination node receives the first dirty page list sent by the source node and sends a request to the source node to read the memory block data corresponding to the first dirty page list.

[0017] The source node sends the memory block data corresponding to the first dirty page list to the destination node;

[0018] The destination node receives the memory block data corresponding to the first dirty page list, updates the first dirty page list and the second dirty page list, and adds the record of the received memory block to the first memory block list and the second memory block list.

[0019] In one possible implementation, the list of data in different storage states includes: a first list of storage blocks recording the data of the persistent storage blocks of the partition basket stored in the original node; a first list of memory blocks recording the memory blocks of the partition basket that have been copied in the original node; and a second list of memory blocks recording the memory blocks of the partition basket that are in the destination node.

[0020] The list of data based on the different storage states migrates the data from the partition basket to the destination data partition, including:

[0021] The destination node receives the list of non-dirty pages sent by the source node and sends a request to the source node to read the memory block data corresponding to the list of non-dirty pages.

[0022] The source node sends the memory block data corresponding to the non-dirty page list to the destination node;

[0023] The destination node receives the memory block data corresponding to the non-dirty page list, updates the first storage block list, and adds the record of the received memory block to the first memory block list and the second memory block list.

[0024] In one possible implementation, the list of data in different storage states includes: a first list of storage blocks recording the data of the persistent storage blocks of the partition basket stored in the original node; a first list of memory blocks recording the memory blocks of the partition basket that have been copied in the original node; and a second list of memory blocks recording the memory blocks of the partition basket that are in the destination node.

[0025] The list of data based on the different storage states migrates the data from the partition basket to the destination data partition, including:

[0026] The destination node receives the first list of storage blocks sent by the origin node and sends a request to the origin node to read the data corresponding to the first list of storage blocks.

[0027] The origin node reads the data corresponding to the first storage block list from persistent storage into memory and sends it to the destination node;

[0028] The destination node receives the memory block data corresponding to the first storage block list, updates the first storage block list, and adds the record of the received memory block to the first memory block list and the second memory block list.

[0029] In one possible implementation, the shadow further includes: a list of third data, representing the data that needs to be read or written but is not yet copied to the destination node;

[0030] After routing the operation on the data basket to the node where the destination data partition is located, the method includes:

[0031] The destination node receives an access request for the basket of partitions to be migrated;

[0032] The destination node records the access request data in the list of the third data;

[0033] The destination node sends an access request to the origin node;

[0034] The original node sends the data corresponding to the access request to the destination node;

[0035] After receiving the data corresponding to the access request, the destination node deletes the corresponding record from the list of third data.

[0036] In one possible implementation, the data in the access request includes data stored only in the persistent storage of the original node;

[0037] The source node sends the data corresponding to the access request to the destination node, including:

[0038] The source node reads the data corresponding to the access request from persistent storage and sends it to the destination node.

[0039] In one possible implementation, the data in the access request includes non-dirty page data that is stored both in the persistent storage of the original node and in memory;

[0040] The source node sends the data corresponding to the access request to the destination node, including:

[0041] The source node reads the data corresponding to the access request from memory and sends it to the destination node.

[0042] In one possible implementation, the data in the access request includes dirty page data that is stored both in the persistent storage of the original node and in memory;

[0043] Before the origin node sends the data corresponding to the access request to the destination node, the method further includes: the origin node determining that the data in the access request is dirty page data;

[0044] The source node sends the data corresponding to the access request to the destination node, including:

[0045] The source node sends the data corresponding to the access request read from memory and the prompt information indicating that the data is dirty page data to the destination node.

[0046] A second aspect of this application provides a data partitioning adjustment system, comprising:

[0047] The replication unit is used to copy the uncommitted access list and concurrency control logic of the partition basket to be migrated from the original node to the destination node; the original node is the node where the original data partition is located; the destination node is the node where the destination data partition is located.

[0048] A shadow creation unit is used to create a shadow of the basket of partitions to be migrated in the destination data partition; the shadow includes a list of data in different storage states;

[0049] The routing unit is used to route access requests to the data basket to the destination node, so that operation requests for the data basket to be migrated during the operation are directed to the shadow.

[0050] The request processing unit is used to enable the destination node to process access requests for the partition basket based on the uncommitted access list and concurrency control logic.

[0051] The data migration unit is used to migrate data from the partition basket to the destination data partition based on the list of data in different storage states.

[0052] Compared to existing technologies, the advantages of this solution are as follows:

[0053] The data partitioning adjustment method provided in this application can dynamically adjust data partitions at runtime, and can allocate computing and storage resources to where they are needed at any time, thereby improving the overall resource utilization of the entire cluster. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 A flowchart of a data partitioning adjustment method provided in this application embodiment;

[0056] Figure 2 This is a schematic diagram of data partition adjustment provided in an embodiment of this application;

[0057] Figure 3 This is a structural diagram of a data partitioning adjustment system provided in an embodiment of this application. Detailed Implementation

[0058] Terminology Explanation:

[0059] Primary Key: The data partitions described in this application should first and foremost belong to the "same type" of data. The criterion for determining whether data belongs to the same type is whether a common field (i.e., attribute) can be extracted. This set of attributes exists in all data of the same type, and the value of this set of attributes can uniquely identify a piece of data. Then this set of attributes is the primary key.

[0060] Data partitioning: Data partitioning assumes that the data is of the same type and a primary key can be found. Under this premise, there are several methods for data partitioning:

[0061] Vertical partitioning: This partitioning method, which places different attributes of similar data into different partitions, is called vertical partitioning.

[0062] Horizontal partitioning: This partitioning method involves placing different records of the same type of data into different partitions.

[0063] Hybrid partitioning: Treating similar data as a two-dimensional table, dividing the two-dimensional table into multiple sub-two-dimensional tables, and placing these sub-two-dimensional tables in different partitions is called hybrid partitioning.

[0064] Partition Basket: A partition basket is the smallest unit of data partitioning. While theoretically, data partitioning algorithms can support partitioning granularity down to a single data element (i.e., an attribute of a data record), in practice, due to the typically large volume of data, to reduce the resource consumption of data partitioning calculations, a batch of data is usually divided into a "basket" based on a calculated value (derived from the data partitioning algorithm). This basket becomes the actual smallest unit of data partitioning. Data within the same basket can only be placed in the same data partition. A data partition can consist of one or more baskets.

[0065] Session: In data processing systems with concurrency control, the concept of a session often exists. A session is typically understood by the data processing system as being associated with a specific end user. Data processing between end users needs to be isolated to some extent to avoid confusion. Furthermore, the operations of a specific end user are sequential and cannot be reversed. These two characteristics together constitute a description of a session.

[0066] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the embodiments of this application will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0067] As mentioned earlier, the computing and storage resources of a single server are limited. Data partitioning algorithms can distribute data across different servers for computation and storage. This can significantly reduce the demands on the computing and storage resources of a single server. However, in data partitioning scenarios, uneven initial partitioning can lead to uneven demands on computing and storage resources. Therefore, after running for a period of time, data partitioning often needs to be readjusted.

[0068] Currently, when data partitions are readjusted, access requests need to be stopped. This prevents data partition readjustment from being performed during peak business periods.

[0069] Scenario 1: After data partitioning, the actual data distribution is uneven.

[0070] When creating data partitions, it's necessary to choose a partitioning method, algorithm, and key. These should typically be attributes that distinguish a batch of data from a common set of attributes, and business implications should be considered to reduce cross-partition join computations. After partitioning, and after a period of operation, it may be discovered that the data partitions are unbalanced, with a large amount of data actually distributed across only a few partitions. The centralized partitions become the performance and capacity bottleneck for the entire cluster. The simplest solution is to migrate some of the data partitions from the centralized partitions to partitions with more available resources. Previously, data partition adjustments could only be performed when business services were stopped. However, actual business operations require 24 / 7 operation, making it more difficult to choose a suitable time window for data partition adjustments.

[0071] Scenario 2: During peak business hours, it was found that a large amount of computation was concentrated in a few data partitions.

[0072] During peak business hours, insufficient cluster performance was found, impacting the quality of business services. However, because the cluster had already undergone data partitioning, simply adding nodes would not increase the cluster's capacity or performance; data partitioning adjustments were required for the new nodes to integrate into the cluster. However, existing methods for adjusting data partitions required stopping business services. But during peak business hours, it was impossible to stop business services. Therefore, most data processing capacity expansions ultimately fail at the data partitioning adjustment step, preventing the timely allocation of resources and commencement of operations.

[0073] During peak business hours, insufficient cluster performance was found, impacting the quality of business services. However, because the cluster had already undergone data partitioning, simply adding nodes would not increase the cluster's capacity or performance; data partitioning adjustments were required for the new nodes to integrate into the cluster. However, existing methods for adjusting data partitions required stopping business services. But during peak business hours, it was impossible to stop business services. Therefore, most data processing capacity expansions ultimately fail at the data partitioning adjustment step, preventing the timely allocation of resources and commencement of operations.

[0074] In view of this, embodiments of this application provide a data partitioning adjustment method and system, which can be applied to cluster management or control platforms. Data partitions can be dynamically adjusted at runtime, and computing and storage resources can be allocated to where needed at any time, thereby improving the overall resource utilization of the entire cluster.

[0075] See Figure 1 The figure is a flowchart of a data partitioning adjustment method according to an embodiment of this application. Figure 1 As shown, the method includes:

[0076] S110. Copy the uncommitted access list and concurrency control logic of the partition basket to be migrated from the original node to the destination node; the original node is the node where the original data partition is located; the destination node is the node where the destination data partition is located.

[0077] Following S110, the method further includes: sending the log records corresponding to the dirty pages of the committed data in memory to the destination node so that the destination node can persist the log records; and the original node stops refreshing the persistent storage of the data basket to be migrated.

[0078] S120. Create a shadow of the basket of partitions to be migrated in the destination data partition; the shadow includes a list of data in different storage states;

[0079] S130. Route access requests to the data basket to the destination node so that operation requests for the data basket to be migrated during the operation are directed to the shadow.

[0080] In one embodiment, S130 includes:

[0081] In the data partition location determination terminal, the node where the basket of the partition to be migrated is located is modified to the destination node; the data partition location determination terminal includes a client and / or a proxy server.

[0082] S140, The destination node processes access requests for the partition basket based on the uncommitted access list and concurrency control logic.

[0083] S150: Based on the list of data in different storage states, migrate the data of the partition basket to be migrated to the destination data partition.

[0084] The data partitioning adjustment method provided in this application embodiment can dynamically adjust data partitions at runtime, and can allocate computing and storage resources to where they are needed at any time, thereby improving the overall resource utilization of the entire cluster.

[0085] In one embodiment, the shadow includes:

[0086] 1. A list of all uncommitted accesses to the partition basket on the original node;

[0087] 2. The access list of the partition basket on the destination node (this list is empty when a new shadow is created);

[0088] 3. A list of the first memory blocks that have been copied from the original node in the partition basket;

[0089] 4. Access requires reading and writing, but is a third data list of data that has not yet been copied to the destination node;

[0090] 5. The first dirty page list of dirty page data stored in the original node's memory block but not in the persistent storage block;

[0091] 6. A second dirty page list containing dirty page data stored in the destination node's memory block but not in the persistent storage block;

[0092] 7. The partition basket stores the first list of persistent data blocks in the original node;

[0093] 8. A list of second storage blocks containing data that has been stored in persistent storage at the destination point;

[0094] 9. List of locks for partitioned baskets;

[0095] 10. The partition basket is already in the second memory block list of the memory blocks in the destination node.

[0096] In one embodiment, S150 includes:

[0097] The destination node sends a request to the origin node to read the memory block data corresponding to the first dirty page list.

[0098] The source node sends the memory block data corresponding to the first dirty page list to the destination node;

[0099] The destination node receives the memory block data corresponding to the first dirty page list, updates the first dirty page list and the second dirty page list, and adds the record of the received memory block to the first memory block list and the second memory block list.

[0100] In one embodiment, S150 includes:

[0101] The destination node sends a request to the source node to read the memory block data corresponding to the non-dirty page list.

[0102] The source node sends the memory block data corresponding to the non-dirty page list to the destination node;

[0103] The destination node receives the memory block data corresponding to the non-dirty page list, updates the first storage block list, and adds the record of the received memory block to the first memory block list and the second memory block list.

[0104] In one embodiment, S150 includes:

[0105] The destination node receives the first list of storage blocks sent by the origin node and sends a request to the origin node to read the data corresponding to the first list of storage blocks.

[0106] The origin node reads the data corresponding to the first storage block list from persistent storage into memory and sends it to the destination node;

[0107] The destination node receives the memory block data corresponding to the first storage block list, updates the first storage block list, and adds the record of the received memory block to the first memory block list and the second memory block list.

[0108] In one embodiment, the shadow further includes: a list of third data, used to represent the data that requires read / write access but is not yet copied to the destination node;

[0109] After routing the operation on the data basket to the node where the destination data partition is located, the method includes:

[0110] The destination node receives an access request for the basket of partitions to be migrated;

[0111] The destination node records the access request data in the list of the third data;

[0112] The destination node sends an access request to the origin node;

[0113] The original node sends the data corresponding to the access request to the destination node;

[0114] After receiving the data corresponding to the access request, the destination node deletes the corresponding record from the list of third data.

[0115] In one embodiment, the data in the access request includes data stored only in the persistent storage of the original node;

[0116] The source node sends the data corresponding to the access request to the destination node, including:

[0117] The source node reads the data corresponding to the access request from persistent storage and sends it to the destination node.

[0118] In one embodiment, the data in the access request includes non-dirty page data that is stored both in the persistent storage of the original node and in memory;

[0119] The source node sends the data corresponding to the access request to the destination node, including:

[0120] The source node reads the data corresponding to the access request from memory and sends it to the destination node.

[0121] In one embodiment, the data in the access request includes dirty page data that is stored both in the persistent storage of the original node and in memory;

[0122] Before the origin node sends the data corresponding to the access request to the destination node, the method further includes: the origin node determining that the data in the access request is dirty page data;

[0123] The source node sends the data corresponding to the access request to the destination node, including:

[0124] The source node sends the data corresponding to the access request read from memory and the prompt information indicating that the data is dirty page data to the destination node.

[0125] See Figure 2 This figure is a schematic diagram of data partitioning provided in an embodiment of this application. Figure 2 As shown, the data partition basket that needs to be migrated is called basket A. The original data partition is called partition A, and the destination data partition is called partition B.

[0126] For example, the process of adjusting the partitions of basket A is as follows:

[0127] Step 1:

[0128] Temporarily retain all accesses to basket A, and copy the list of uncommitted access operations and concurrency control for accessing basket A from the node where partition A is located to the node where partition B is located. Log the changes.

[0129] The committed data in basket A, but which are still dirty pages in memory, including the log records from the last persistent storage synchronization to the current state, are also sent to partition B node. Partition B node immediately persists the received logs.

[0130] For partition A node, stop flushing dirty pages from basket A to the persistent storage of node A.

[0131] Step Two:

[0132] Create a shadow of basket A in partition B, and redirect all operations on basket A to the shadow of basket A.

[0133] It's usually possible to determine which data partition the data operation actually occurred on, with two possible locations: the client and / or the service routing node (also known as a proxy server).

[0134] If this judgment is only made on the service routing node, then on the service routing node that makes this judgment, the node and data partition corresponding to basket A can be modified.

[0135] If this judgment is only performed on the client side, the node and data partition corresponding to basket A for all clients need to be modified. If the client does not support reverse modification, then the node containing partition A needs to create service routing capabilities to route access to basket A to partition B.

[0136] If both the client and the service routing node perform this check, then the steps described above can be executed respectively. However, the modification of the service routing node must precede the modification of the client.

[0137] The shadow of Basket A consists of the following parts:

[0138] 1. All uncommitted access lists of basket A on partition A node.

[0139] 2. The access list of basket A on partition B node (this list is empty when a new shadow is created).

[0140] 3. Basket A has copied a list of memory blocks.

[0141] 4. Access a list of memory blocks that require read / write access but have not yet been copied to partition B node.

[0142] 5. The list of dirty pages in partition A (memory blocks not yet written to persistent storage).

[0143] 6. The list of dirty pages of basket A in partition B node.

[0144] 7. A list of persistent storage blocks for basket A on partition A node.

[0145] 8. The list of persistent storage blocks that basket A has written to partition B node.

[0146] 9. Basket A's lock list

[0147] 10. List of memory blocks of basket A on partition node B

[0148] Except for 1 and 8, the other parts are empty when the shadow is created.

[0149] Step 3:

[0150] Release the operation holding for basket A. At this point, due to step two, these operations will be rerouted to the node where partition B is located for execution.

[0151] Step 4 - Concurrency 1:

[0152] Partition A reads the dirty page list of basket A and sends it to partition B. Partition B, based on the dirty page list, sends memory block read requests to partition A one by one. Partition A sends the requested memory block data (the current state in memory) to partition B. Upon receiving this data, partition B deletes the memory block record from shadow.5 and adds the memory block record to shadow.3, shadow.6, and shadow.10.

[0153] This is a low-priority background operation.

[0154] Step 5 - Concurrency 1:

[0155] Partition A reads a list of pages in memory that are already in memory but are not dirty pages, and sends it to partition B. Partition B then sends memory block read requests to partition A one by one based on this list. Partition A sends the requested memory block data (its current state in memory) to partition B. After receiving the data, partition B deletes the memory block record from shadow.7 and adds the memory block record to shadow.3 and shadow.10.

[0156] This is a low-priority background operation.

[0157] Step Six - Concurrency 1:

[0158] Partition A node reads the list of persistent storage blocks containing data from basket A on partition A node and sends it to partition B node. Partition B node then sends read requests to partition A node for each data block based on this list. Partition A node sends the requested data (which needs to be read from persistent storage first) to partition B node. After receiving the data, partition B node deletes the record for that memory block from shadow.7 and adds the record to shadow.3 and shadow.10.

[0159] This is a low-priority background operation.

[0160] Step 4 - Concurrency 2:

[0161] Partition B node accepts access requests for basket A. First, according to the concurrency control mechanism, it reads all uncommitted access lists and locks for basket A on partition A node to determine if the access request can be executed. If it can, it executes according to the concurrency control requirements. At this point, the data in basket A has the following states, which need to be processed separately:

[0162] (1) This data is in the persistent storage of partition A node, but not in the memory of partition A node, nor on partition B node. At this time, partition B node needs to record the list of access blocks that need to be read and written but have not yet been copied to the memory of partition B node, and initiate a high-priority access request to partition A node (this request has the same priority as other client requests accepted by partition A node, and is higher in priority than the request in step four - concurrency 1). Partition A node reads the data from persistent storage and sends it to partition B node. After receiving the data, partition B node deletes the corresponding record in the list of access blocks that need to be read and written but have not yet been copied to the memory of partition B node. Then it records the list of memory blocks that have been copied by basket A, the list of memory blocks of basket A on partition B node, and the list of dirty pages of basket A on partition B node. There is a persistent storage read / write process on partition B node, which is responsible for finding a suitable time to flush dirty pages to the persistent storage of partition B node.

[0163] (2) This data exists in both the memory and persistent storage of partition A node, but it is not a dirty page. The operation is basically the same as (1), the only difference being that partition A node does not need to read the data from persistent storage.

[0164] (3) This data exists in both memory and persistent storage on partition A node, and it is a dirty page. At this time, partition B node needs to record the list of memory blocks that need to be read and written but have not yet been copied to partition B node, and sends a high-priority access request to partition A node. After receiving the request, A node determines that the page is a dirty page. It then sends this data to partition B node and indicates that the data is dirty page data. At this time, there are two possibilities: 1. Partition A node has already sent its own list of dirty pages to B, then B deletes the memory block record of shadow .5 and adds the records of shadow .3, shadow .6 and shadow .10.

[0165] 2. If partition A node has not yet sent its own dirty page list to B, then A shall delete the memory block record of shadow .5, and B shall be responsible for adding the records of shadow .3, shadow .6 and shadow .10.

[0166] Based on Shadow.1, Shadow.8, and the newly received data processing requests, B decides how to process the data according to the principles of concurrent access control.

[0167] (4) This data has been sent to partition B node. Then, partition B node, based on shadow .1, shadow .8 and the newly received data processing request, decides how to process the data according to the principle of concurrent access control.

[0168] Step 4 - Concurrency 3:

[0169] Partition B node, the background persistent storage read / write process, is responsible for writing the data already sent to partition B node into persistent storage. Priority, from highest to lowest, is as follows:

[0170] (1) The data in partition A is the dirty page data.

[0171] (2) There are no dirty pages in partition A, nor are there any data in partition B that have become dirty due to new access requests.

[0172] (3) Add dirty page data to partition B (this step is not one of the criteria for completing step four - concurrency 3)

[0173] Step Seven:

[0174] Once concurrency one, concurrency two, and concurrency three are all completed, proceed to step seven.

[0175] At this point, the shadow lists 4, 5, and 7 are empty, while the shadow lists 3 and 10 contain all the data blocks of basket A.

[0176] At this point, hold all access to partition B node. Merge shadow .1 and shadow .2 into the access request list of partition B node, merge shadow .9 into the lock list of partition B node, merge shadow .6 into the dirty page list of partition B node, merge shadow .3 and shadow .10 into the memory data list of partition B node, and merge shadow .8 into the persistent data list of partition B node.

[0177] Step 8:

[0178] Release the access hold for partition B node.

[0179] At this point, the entire migration process is complete.

[0180] Special Scenario 1:

[0181] During the migration process, new data needs to be added, which requires the creation of new data blocks. Since this operation is handled by partition B node, partition B node will add a new memory block record in shadow.10. All other data additions are processed normally.

[0182] Special Scenario Two:

[0183] During the migration process, partition A node and / or partition B node may fail. Steps 1 and 2 are particularly noteworthy. Since the actual data migration has not yet begun in steps 1 and 2, the failure of either partition A or partition B node will immediately halt the data partition adjustment. The service route modification will be rolled back, and access to basket A will be removed.

[0184] After step three, until the adjustment operation is complete, because the logs of the committed operations on node A have been written to both A and B, regardless of whether node A or node B fails, the dirty pages on A can be redone based on the committed logs after restarting. This process occurs on A. Once the redo on A is complete, the entire basket A is sent to B, and B continues to redo the committed operations that occurred on node B. When all redoes are complete, B checks the logs of basket A and rolls back all uncommitted operations. After the rollback is complete, B migrates basket A according to the offline migration process. Service is restored after all operations are completed.

[0185] This application provides a data partitioning adjustment system, including:

[0186] The copying unit 310 is used to copy the uncommitted access list and concurrency control logic of the partition basket to be migrated from the original node to the destination node; the original node is the node where the original data partition is located; the destination node is the node where the destination data partition is located.

[0187] The shadow creation unit 320 is used to create a shadow of the basket of partitions to be migrated in the destination data partition; the shadow includes a list of data in different storage states;

[0188] The routing unit 330 is used to route access requests to the data basket to the destination node, so that operation requests for the data basket to be migrated during the operation are directed to the shadow.

[0189] The request processing unit 340 is used to enable the destination node to process access requests for the partition basket based on the uncommitted access list and concurrency control logic.

[0190] The data migration unit 350 is used to migrate data from the partition basket to the destination data partition based on the list of data in different storage states.

[0191] It should be noted that the term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description.

[0192] It should be noted that the concepts of "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0193] It should be noted that although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0194] While several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0195] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A data partitioning adjustment method, characterized in that, include: Copy the uncommitted access list and concurrency control logic of the partition basket to be migrated from the original node to the destination node; The original node is the node where the original data partition is located; The destination node is the node where the destination data partition is located; Create a shadow of the basket of partitions to be migrated in the destination data partition; the shadow includes a list of data in different storage states; Access requests to the data basket are routed to the destination node so that operation requests for the data basket to be migrated during the operation are directed to the shadow. The destination node processes access requests for the partition basket based on the uncommitted access list and concurrency control logic. Based on the list of data in different storage states, the data in the partition basket to be migrated is migrated to the destination data partition; The list of data in different storage states includes: a first dirty page list that records dirty page data of the partition basket stored in the memory block of the origin node but not in the persistent storage block; a second dirty page list that records dirty page data of the partition basket stored in the memory block of the destination node but not in the persistent storage block; a first memory block list that records the memory block of the partition basket that has been copied in the origin node; and a second memory block list that records the memory block of the partition basket that is in the destination node. The list of data based on the different storage states migrates the data from the partition basket to the destination data partition, including: The destination node receives the first dirty page list sent by the source node and sends a request to the source node to read the memory block data corresponding to the first dirty page list. The source node sends the memory block data corresponding to the first dirty page list to the destination node; The destination node receives the memory block data corresponding to the first dirty page list, updates the first dirty page list and the second dirty page list, and adds the record of the received memory block to the first memory block list and the second memory block list.

2. The method according to claim 1, characterized in that, The process of routing access requests to the data basket to the destination node includes: In the data partition location determination terminal, the node where the basket of the partition to be migrated is located is modified to the destination node; the data partition location determination terminal includes a client and / or a proxy server.

3. The method according to claim 1, characterized in that, The list of data in different storage states includes: a first storage block list recording the data of the persistent storage blocks where the partition basket is stored in the original node; a first memory block list recording the memory blocks in which the partition basket has been copied in the original node; and a second memory block list recording the memory blocks in which the partition basket is in the destination node. The list of data based on the different storage states migrates the data from the partition basket to the destination data partition, including: The destination node receives the list of non-dirty pages sent by the source node and sends a request to the source node to read the memory block data corresponding to the list of non-dirty pages. The source node sends the memory block data corresponding to the non-dirty page list to the destination node; The destination node receives the memory block data corresponding to the non-dirty page list, updates the first storage block list, and adds the record of the received memory block to the first memory block list and the second memory block list.

4. The method according to claim 1, characterized in that, The list of data in different storage states includes: a first storage block list recording the data of the persistent storage blocks where the partition basket is stored in the original node; a first memory block list recording the memory blocks in which the partition basket has been copied in the original node; and a second memory block list recording the memory blocks in which the partition basket is in the destination node. The list of data based on the different storage states migrates the data from the partition basket to the destination data partition, including: The destination node receives the first list of storage blocks sent by the origin node and sends a request to the origin node to read the data corresponding to the first list of storage blocks. The origin node reads the data corresponding to the first storage block list from persistent storage into memory and sends it to the destination node; The destination node receives the memory block data corresponding to the first storage block list, updates the first storage block list, and adds the record of the received memory block to the first memory block list and the second memory block list.

5. The method according to claim 1, characterized in that, The shadow also includes: a list of third data, representing data that requires read / write access but has not yet been copied to the destination node; After routing the operation on the data basket to the node where the destination data partition is located, the method includes: The destination node receives an access request for the basket of partitions to be migrated; The destination node records the access request data in the list of the third data; The destination node sends an access request to the origin node; The original node sends the data corresponding to the access request to the destination node; After receiving the data corresponding to the access request, the destination node deletes the corresponding record from the list of third data.

6. The method according to claim 5, characterized in that, The data in the access request includes data that is stored only in the persistent storage of the original node; The source node sends the data corresponding to the access request to the destination node, including: The source node reads the data corresponding to the access request from persistent storage and sends it to the destination node.

7. The method according to claim 5, characterized in that, The data in the access request includes non-dirty page data that is stored both in the persistent storage of the original node and in memory; The source node sends the data corresponding to the access request to the destination node, including: The source node reads the data corresponding to the access request from memory and sends it to the destination node.

8. The method according to claim 5, characterized in that, The data in the access request includes dirty page data that is stored both in the persistent storage of the original node and in memory. Before the origin node sends the data corresponding to the access request to the destination node, the method further includes: the origin node determining that the data in the access request is dirty page data; The source node sends the data corresponding to the access request to the destination node, including: The source node sends the data corresponding to the access request read from memory and the prompt information indicating that the data is dirty page data to the destination node.

9. A data partitioning adjustment system, characterized in that, include: The replication unit is used to copy the uncommitted access list and concurrency control logic of the partition basket to be migrated from the source node to the destination node; The original node is the node where the original data partition is located; The destination node is the node where the destination data partition is located; A shadow creation unit is used to create a shadow of the basket of partitions to be migrated in the destination data partition; the shadow includes a list of data in different storage states; The routing unit is used to route access requests to the data basket to the destination node, so that operation requests for the data basket to be migrated during the operation are directed to the shadow. The request processing unit is used to enable the destination node to process access requests for the partition basket based on the uncommitted access list and concurrency control logic. The data migration unit is used to migrate the data of the partition basket to be migrated to the destination data partition based on the list of data in different storage states. The list of data in different storage states includes: a first dirty page list recording dirty page data of the partition basket stored in the memory block of the origin node but not in the persistent storage block; a second dirty page list recording dirty page data of the partition basket stored in the memory block of the destination node but not in the persistent storage block; a first memory block list recording the memory block of the partition basket that has been copied in the origin node; and a second memory block list recording the memory block of the partition basket that is in the destination node. The data migration unit is configured to, in response to the destination node receiving the first dirty page list sent by the origin node, initiate a request to the origin node to read the memory block data corresponding to the first dirty page list; the origin node sends the memory block data corresponding to the first dirty page list to the destination node; the destination node receives the memory block data corresponding to the first dirty page list, updates the first dirty page list and the second dirty page list, and adds the record of the received memory block to the first memory block list and the second memory block list.

Images