A method and device for data redistribution

Abstract

The embodiment of the present invention provides a data redistribution method and device, the data redistribution method comprising: responding to a data redistribution request sent by a computing node, creating a rule table corresponding to the request according to the original table information; generating rule table data and sending it to a storage node corresponding to the rule table data; importing the rule table data into the rule table; completing the rule table data according to the original table; accessing the data in the rule table manner. By generating a new rule table, there is no need to migrate the original data, which improves the efficiency of data redistribution and effectively reduces the impact of data migration during the redistribution process.

Description

Technical Field

[0001] The embodiments of the present invention relate to, but are not limited to, the field of databases, and specifically, relate to, but are not limited to, a control method and device for data redistribution. Background Art

[0002] Currently, the databases used have high concurrency, fast data growth, and 7*24 hours service. As the business expands, the expansion of data may increase by orders of magnitude, so distributed databases with good horizontal expansion capabilities are widely used. Distributed databases generally adopt a shared nothing architecture, that is, each node has independent storage, and the nodes do not share storage. Data is distributed to multiple nodes according to rules, and the nodes are generally connected through optical fiber and other networks.

[0003] Distributed databases have good scalability and can flexibly distribute node data according to business scenarios. For example, as the amount of data increases, there are further requirements for data storage space and query. It is necessary to expand the capacity of the original distributed database, add new data nodes to the distributed database, and migrate data.

[0004] There are two main methods for data redistribution in the industry:

[0005] 1. Lock the table to be redistributed, and do not perform read and write operations until the data redistribution in the table is complete. Depending on the amount of data and resource constraints, data redistribution may take several hours or even days, which may cause the business to be unable to operate the table being redistributed for a long time, resulting in serious impact.

[0006] 2. A more common method is to create a new table, export all the data stored in the old table, import all the data into the new table according to the new rules, and then append incremental data. After completion, lock the old table and switch to the new table. This processing method is inefficient, has high network pressure, requires high storage space, and the process of appending incremental data presents different time spans depending on the volume of business, and there may even be problems of not being able to catch up.

[0007] When the database system is running, the two existing redistribution methods mentioned above have relatively high requirements on the existing network environment, and at the same time, they also require higher emergency handling capabilities of the operation and maintenance operators.

[0008] In addition to capacity expansion, distributed database redistribution scenarios also include capacity reduction, modification of distribution methods, modification of distribution fields, and other scenarios. These scenarios are often faced by database operation and maintenance personnel, and are also functions that mature database products should have. Summary of the invention

[0009] The data redistribution method and device provided by the embodiments of the present invention solve the problems of low efficiency and high overhead of data redistribution processing methods in the prior art.

[0010] In order to solve the above technical problems, an embodiment of the present invention provides a data redistribution method, comprising:

[0011] In response to the data redistribution request sent by the computing node, a rule table corresponding to the request is created based on the original table information; the rule table data is generated and sent to the storage node corresponding to the rule table data; the rule table data is imported into the rule table; the rule table data is completed according to the original table; and the data is accessed in the manner of the rule table.

[0012] Furthermore, the rule table includes sharding keys and metadata information.

[0013] Furthermore, the generation of rule table data includes: exporting distribution key value and node number information, and splitting into files corresponding to storage nodes according to the distribution rules contained in the request; recording the sharding key and distribution mode, as well as the INSERT / UPDATE / DELETE statements successfully executed on the corresponding sharding key.

[0014] Furthermore, the sharding keys of the rule table and the original table are compared with the sharding keys recorded in the INSERT / UPDATE / DELETE statements. If the rule table lacks data corresponding to the sharding key, the data corresponding to the sharding key in the original table is inserted; if the rule table has extra data corresponding to the sharding key, the data corresponding to the sharding key is deleted; when the data corresponding to the sharding key in the rule table is inconsistent with the corresponding data in the original table, it is modified to the data corresponding to the sharding key in the original table.

[0015] Furthermore, before completing the data of the rule table according to the original table, the method further includes: adding a read lock to the table requiring data redistribution; and after accessing the data according to the rule table, the method further includes: releasing the read lock.

[0016] An embodiment of the present invention further provides a data redistribution device, comprising:

[0017] The processor is used to respond to the data redistribution request sent by the computing node and create a rule table corresponding to the request according to the original table information; the storage node is used to generate the rule table data and send it to the storage node corresponding to the rule table data; import the rule table data into the rule table; complete the rule table data according to the original table; the computing node is used to access the data in the manner of the rule table.

[0018] Furthermore, the rule table includes sharding keys and metadata information.

[0019] Furthermore, the storage node is also used to export the distribution key value and node number information, and split it into files corresponding to the storage nodes according to the distribution rules contained in the request; the computing node is also used to record the sharding key and distribution mode, as well as the INSERT / UPDATE / DELETE statements successfully executed on the corresponding sharding key.

[0020] Furthermore, the storage node is also used to compare the sharding keys of the rule table and the original table with the sharding keys recorded in the INSERT / UPDATE / DELETE statement. If the rule table lacks data corresponding to the sharding key, the data corresponding to the sharding key in the original table is inserted; if the rule table has extra data corresponding to the sharding key, the data corresponding to the sharding key is deleted; when the data corresponding to the sharding key in the rule table is inconsistent with the corresponding data in the original table, it is modified to the data corresponding to the sharding key in the original table.

[0021] Furthermore, the computing node is also used to add a read lock to the table that requires data redistribution before the storage node completes the data; and release the read lock after accessing the data in accordance with the rule table.

[0022] An embodiment of the present invention further provides a data redistribution system, comprising the data redistribution device as described above.

[0023] According to the data redistribution method and device provided by the embodiment of the present invention, by generating a new rule table, the original data does not need to be migrated, thereby improving the efficiency of data redistribution and effectively reducing the impact of data migration during the redistribution process. The fields of the rule table only include sharding keys and metadata information, which requires very little storage space, reduces system overhead, and further improves efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of a method for data redistribution provided in Embodiment 1 of the present invention;

[0025] Figure 2 A flowchart of a method for data redistribution provided in Embodiment 2 of the present invention;

[0026] Figure 3 A schematic diagram of program modules of a data redistribution device provided in Embodiment 3 of the present invention;

[0027] Figure 4 A schematic diagram of program modules of a data redistribution device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the following is a further detailed description of the embodiments of the present invention through specific implementation methods combined with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0029] Embodiment 1:

[0030] In order to solve the problem of low efficiency caused by establishing a new table in the prior art, the core idea of ​​this embodiment is to create a rule table according to the new sharding rule after receiving a data redistribution request. The rule table stores the sharding key and metadata information. The so-called metadata information refers to the node information corresponding to the sharding key data. For example, if there are 3 nodes, they can be represented by 1, 2, and 3 respectively. Before the redistribution task is completed, the computing nodes of the distributed database access the data in the same way as the old table. After the redistribution task is completed, the computing nodes access the data according to the rule table.

[0031] See also Figure 1 The data redistribution method involved in this embodiment includes:

[0032] Step 102: In response to the data redistribution request sent by the computing node, a rule table corresponding to the request is created according to the original table information.

[0033] Step 104: Generate rule table data and send it to the storage node corresponding to the rule table data.

[0034] Step 106: Import the rule table data into the rule table.

[0035] Step 108: Complete the data of the rule table according to the original table.

[0036] Step 110: Access data in accordance with the rule table.

[0037] The data redistribution method provided in the first embodiment of the present invention improves the efficiency of data redistribution and reduces the impact of data migration by creating a rule table instead of migrating data by creating a new table.

[0038] Embodiment 2:

[0039] See also Figure 2 The data redistribution method involved in this embodiment may include:

[0040] Step 202: Create a rule table.

[0041] After initiating a data redistribution request, a new rule table is created based on the original table information. The sharding key of the new rule table is consistent with the sharding method of the original table in the redistribution request. For example, if the original table is hashed to 3 nodes according to the field col1, and the redistribution requires hashing to 4 nodes, then the rule table is hashed to 4 nodes according to the field col1.

[0042] In this embodiment, the redistribution of the distribution mode from 3 nodes to 4 nodes is used as an example for explanation. The original table name is tbs_info_detail, the distribution mode is hash(uuid)(g1,g2,g3), and the processor creates a hidden rule table tbs_info_detail_res_rule, and the distribution mode is hash(uuid)(g1,g2,g3,g4). There are two fields in the rule table, uuid and groupid. The attributes of the uuid field are exactly the same as those of the original table, while the attributes of the groupid field can use the unsignedtinyint attribute (only one byte). At the same time, the rule needs to create an index for uuid (if the uuid of the original table is the primary key, create a primary key for uuid).

[0043] Step 204: Generate rule table data.

[0044] Each storage node contains a redistribution control module, which is used to generate and import metadata rule table data. These redistribution control modules export distribution key values ​​from each storage node in parallel, and carry storage node number information. The split files are split into files of different storage nodes according to the new distribution rules and transmitted to the corresponding storage nodes.

[0045] In this step, all INSERT / UPDATE / DELETE statements involving the shard key and executed successfully are recorded, and the shard key value and the corresponding groupid are recorded.

[0046] Step 206: Import data into the rule table.

[0047] The redistribution control module of each storage node can import the transmitted data into the rule table.

[0048] Step 208: Add a read lock.

[0049] Notify the computing nodes to add read locks to the tables that need to be redistributed.

[0050] Step 210: Complete the data.

[0051] Compare the shard key values ​​recorded in the INSERT / UPDATE / DELETE statements involving the shard key with the shard keys in the original table and the rule table. If the rule table lacks the data corresponding to the shard key value, insert the data corresponding to the shard key value in the original table. If the rule table has more data corresponding to the shard key value, delete the data corresponding to the shard key value. If the data in the rule table is inconsistent with the original table, modify it to be consistent with the original table.

[0052] Step 212: Modify the access mode.

[0053] Notify the computing nodes to modify the access method of the table, modify the computing node's own computing distribution algorithm to access the rule table. The INSERT / UPDATE / DELETE statements involving the table need to modify the rule table at the same time, and the new data is inserted into the new data node first.

[0054] Step 214: Release the read lock.

[0055] Releases the read lock.

[0056] At this point, the redistribution task has been completed.

[0057] Step 216: Align distribution rules.

[0058] To solve the problem of inconsistency between the rule table distribution method and the computing nodes, you can run the batches in idle time, for example:

[0059] 1. First, query the rows with metadata field 2 in the rule table from node 1, migrate this part of data to node 2 in batches, and change the rule table at the same time.

[0060] 2. When most of the data is successfully migrated, lock the table and migrate the remaining data;

[0061] 3. Notify the computing nodes to change the access method of the table;

[0062] 4. Clean up the rule table.

[0063] The data redistribution method provided in the second embodiment of the present invention creates a rule table, and the rule table only includes two fields, thereby improving the redistribution efficiency and reducing the impact of data migration, while further reducing the redistribution overhead.

[0064] Embodiment three:

[0065] See also Figure 3 The data redistribution device 300 involved in this embodiment includes:

[0066] Computing node 301, processor 302 and storage node 303. The processor 302 is used to respond to the data redistribution request sent by the computing node 301, and create a rule table corresponding to the request according to the original table information. There are multiple storage nodes 303, which are used to generate rule table data and send it to the storage node 303 corresponding to the rule table data, import the rule table data into the rule table, and complete the rule table data according to the original table. The computing node 301 is used to access data in the way of the rule table.

[0067] The data redistribution device provided in the third embodiment of the present invention improves the efficiency of data redistribution and reduces the impact of data migration by creating a rule table instead of migrating data by creating a new table.

[0068] Embodiment 4:

[0069] See also Figure 4 The data redistribution device 400 involved in this embodiment includes:

[0070] Computing node 401, processor 402 and storage node 403. There are four storage nodes 403.

[0071] After computing node 401 initiates a data redistribution request, processor 402 creates a new rule table based on the original table information, and the sharding key of the new rule table is consistent with the sharding method of the original table of the redistribution request.

[0072] Each of the four storage nodes 403 includes a redistribution control module 403a, which is used to generate and import metadata rule table data. These redistribution control modules 403a derive distribution key values ​​from each storage node 403 in parallel, and carry storage node number information. The split files are split into files of different storage nodes 403 according to the new distribution rules, and are transmitted to the corresponding storage nodes 403. The computing node 401 records all INSERT / UPDATE / DELETE statements involving the shard key and successfully executed, and records the shard key value and the corresponding groupid.

[0073] The redistribution control module 403a of the storage node 403 imports the transmitted data into the rule table.

[0074] Processor 402 notifies computing node 401 to add a read lock to the table that needs to be redistributed.

[0075] The redistribution control module 403a compares the shard key value recorded in the INSERT / UPDATE / DELETE statement involving the shard key recorded by the computing node 401 with the shard keys in the original table and the rule table. If the rule table lacks the data corresponding to the shard key value, the data corresponding to the shard key value in the original table is inserted. If the rule table has more data corresponding to the shard key value, the data corresponding to the shard key value is deleted. If the data in the rule table is inconsistent with the original table, it is modified to be consistent with the original table.

[0076] Processor 402 notifies computing node 401 to modify the access method of the table, and modifies the computing distribution algorithm of computing node 401 itself to access according to the rule table. INSERT / UPDATE / DELETE statements involving the table need to modify the rule table at the same time, and new data is inserted into the newly added data node first.

[0077] The computing node 401 releases the added read lock. At this point, the redistribution task has been completed.

[0078] If it is necessary to distribute rules, proceed according to step 216 of the second embodiment.

[0079] The data redistribution device provided in the fourth embodiment of the present invention creates a rule table, and the rule table only includes two fields, thereby improving the redistribution efficiency and reducing the impact of data migration, while further reducing the redistribution overhead.

[0080] The present invention also provides a data redistribution system, including the data redistribution device of the fourth embodiment, which can efficiently execute data redistribution tasks.

[0081] It can be seen that those skilled in the art should understand that all or some of the steps, systems, and functional modules / units in the above disclosed methods can be implemented as software (which can be implemented with computer program code executable by a computing device), firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component can have multiple functions, or a function or step can be performed by several physical components in cooperation. Some or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit.

[0082] In addition, it is well known to those skilled in the art that communication media generally contain computer readable instructions, data structures, computer program modules or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery media. Therefore, the present invention is not limited to any specific hardware and software combination.

[0083] The above contents are further detailed descriptions of the embodiments of the present invention in combination with specific implementation methods, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A data redistribution method, comprising: In response to a data redistribution request sent by a computing node, a rule table corresponding to the request is created according to the original table information; Generate rule table data and send it to the storage node corresponding to the rule table data; Importing the rule table data into the rule table; Completing the data of the rule table according to the original table; Accessing data in accordance with the rule table; The rule table includes a shard key and metadata information, and the metadata information is node information corresponding to the shard key data.

2. The data redistribution method according to claim 1, characterized in that: The generation rule table data includes: Export the distribution key value and node number information, and split them into files corresponding to the storage nodes according to the distribution rules contained in the request; Record the sharding key and distribution mode, as well as the INSERT / UPDATE / DELETE statements that are successfully executed on the corresponding sharding key.

3. The data redistribution method according to claim 2, characterized in that: The data of the rule table supplemented according to the original table includes: The sharding keys of the rule table and the original table are compared with the sharding keys recorded in the INSERT / UPDATE / DELETE statements. If the rule table lacks data corresponding to the sharding key, the data corresponding to the sharding key in the original table is inserted. If the rule table has extra data corresponding to the sharding key, the data corresponding to the sharding key is deleted. When the data corresponding to the sharding key in the rule table is inconsistent with the corresponding data in the original table, it is modified to the data corresponding to the sharding key in the original table.

4. The data redistribution method according to any one of claims 1 to 3, characterized in that: Before completing the data of the rule table according to the original table, the following steps are also included: Add a read lock to the table that requires data redistribution; After accessing the data in the manner of the rule table, the method further includes: releasing the read lock.

5. A data redistribution device, comprising: A processor, configured to respond to a data redistribution request sent by a computing node and create a rule table corresponding to the request according to original table information; A storage node, used for generating rule table data and sending the data to a storage node corresponding to the rule table data; Importing the rule table data into the rule table; Completing the data of the rule table according to the original table; A computing node, used to access data in accordance with the rule table; The rule table includes a shard key and metadata information, and the metadata information is node information corresponding to the shard key data.

6. The data redistribution device according to claim 5, characterized in that: The rule table includes a shard key and metadata information.

7. The data redistribution device according to claim 5, characterized in that: The storage node is further used to export the distribution key value and node number information, and split the files into corresponding storage nodes according to the distribution rule included in the request; The computing node is also used to record the sharding key and distribution mode, as well as the INSERT / UPDATE / DELETE statements that are successfully executed on the corresponding sharding key.

8. The data redistribution device according to claim 7, characterized in that: The storage node is further used to compare the sharding keys of the rule table and the original table with the sharding keys recorded in the INSERT / UPDATE / DELETE statements. If the rule table lacks data corresponding to the sharding key, the data corresponding to the sharding key in the original table is inserted; if the rule table has extra data corresponding to the sharding key, the data corresponding to the sharding key is deleted; when the data corresponding to the sharding key in the rule table is inconsistent with the corresponding data in the original table, it is modified to the data corresponding to the sharding key in the original table.

9. The data redistribution device according to any one of claims 5 to 8, characterized in that: The computing node is further used to add a read lock to a table that requires data redistribution before the storage node completes the data; and release the read lock after accessing the data in accordance with the rule table.

10. A data redistribution system, characterized in that: It comprises a data redistribution device as claimed in any one of claims 5 to 9.

Images