Method and apparatus for data reallocation

Abstract

This application discloses a method and apparatus for data reallocation, the method including the steps of: in response to a data reallocation request sent from a computing node (301), creating a rule table corresponding to the request based on information in an original table; generating rule table data and sending it to a storage node (303) corresponding to the rule table data; importing the rule table data into the rule table, complementing the rule table data according to the original table, and accessing data according to the method of the rule table.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is filed based on a Chinese patent application bearing application number "202010600815.9" and filed on June 28, 2020, and claims priority to that Chinese patent application, the entire text of which is incorporated herein by reference. [Technical field]

[0002] FIELD OF THE DISCLOSURE Embodiments of the present application relate to the field of databases, and in particular to a method and apparatus for data reallocation. [Background technology]

[0003] There are two main methods of data reallocation in the industry:

[0004] 1. Lock the tables that need to be redistributed and prevent read and write operations from occurring until all data in the tables has been redistributed. Depending on the amount of data and resource limitations, data redistribution can take hours or even days, which can have significant impact as the service cannot operate on the tables being redistributed for an extended period of time.

[0005] 2. A more common method is to create a new table, export all the data stored in the old table, then import it all into the new table according to the new rules, then add the incremental data, and when it is complete, lock the old table and switch to using the new table. This processing method is inefficient, has a high network load, and high storage requirements. In addition, the process of tracking the incremental data will have different time spans depending on the amount of service, and may never be able to keep up.

[0006] When using the above two conventional reallocation methods during the operation of a database system, the requirements for the existing network environment are relatively high, and the requirements for the emergency response capabilities of operation and maintenance personnel are even higher.

[0007] In addition to capacity expansion, distributed database redistribution scenarios include capacity reduction, changes in distribution methods, and changes in distribution fields. These are scenarios that database operators and maintainers frequently face, and are also functions that mature database products should have. Summary of the Invention [Means for solving the problem]

[0008] According to an embodiment of the present application, a data reallocation method is provided, which includes the steps of: in response to a data reallocation request sent from a computing node, creating a rule table corresponding to the request based on information of an original table; generating rule table data and sending it to a storage node corresponding to the rule table data; importing the rule table data into the rule table; complementing the rule table data to match the original table; and accessing data according to a method of the rule table.

[0009] An embodiment of the present application further provides an apparatus for data reallocation, comprising: a processor configured to, in response to a data reallocation request sent from a computing node, create a rule table corresponding to the request based on information of an original table; a storage node configured to generate rule table data, send the rule table data to a storage node corresponding to the rule table data, import the rule table data into the rule table, and complement the rule table data to match the original table; and a computing node configured to access data according to a manner of the rule table.

[0010] An embodiment of the present application further provides a data reallocation system including a data reallocation device as described above.

[0011] According to an embodiment of the present application, there is further provided an apparatus, comprising one or more processors and a memory storing one or more programs, the one or more programs, when executed by the one or more processors, causing the one or more processors to implement the above data reallocation method.

[0012] An embodiment of the present application further provides a storage medium, the storage medium storing a computer program, which, when executed by a processor, implements the data reallocation method described above. [Brief description of the drawings]

[0013] [Figure 1] 1 is a flowchart of a data reallocation method provided by Embodiment 1 of the present application; [Diagram 2] 4 is a flowchart of a data reallocation method provided by Embodiment 2 of the present application; [Diagram 3] FIG. 11 is a program module schematic diagram of a data reallocation device provided in embodiment 3 of the present application. [Figure 4] FIG. 11 is a program module schematic diagram of a data reallocation device provided in embodiment 4 of the present application. [Diagram 5] FIG. 1 is a schematic diagram of a configuration of an apparatus provided according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] In order to make the objectives, technical solutions and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with specific embodiments combined with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to interpret the present application, and are not used to limit the present application.

[0015] Currently applied databases have high concurrency, rapid data growth, and provide 24 / 7 service. As business expands, data expansion can be orders of magnitude larger, so distributed databases with excellent scale-out capabilities are widely used. Distributed databases generally adopt a share-nothing architecture, that is, each node has independent storage, no storage is shared between nodes, data is distributed to multiple nodes according to rules, and nodes are generally connected via a network such as optical fiber.

[0016] Distributed databases have good scalability and can flexibly distribute node data according to business scenarios. For example, as the data volume order grows, more requirements are placed on data storage and query, which requires expanding the capacity of the existing distributed database, adding new data nodes to the distributed database, and performing data migration.

[0017] The data reallocation method and apparatus provided by the embodiments of the present application solves the problem of low efficiency and large overhead of data reallocation processing methods in some cases.

[0018] EMBODIMENT 1

[0019] In order to solve the efficiency loss caused by creating a new table in some situations, the core idea of ​​this embodiment is to create a rule table to hold a sharding key and metadata information according to a new sharding rule when a data reallocation request is received, and the metadata information is the node information corresponding to the sharding key data, for example, there are three nodes which can be represented as 1, 2, and 3 respectively. Until the reallocation task is completed, the computing nodes of the distributed database access according to the manner of the old table, and when the reallocation task is completed, the computing nodes access data according to the rule table.

[0020] Referring to FIG. 1, the data reallocation method according to the embodiment includes the following steps.

[0021] In step 102, in response to a data reallocation request sent from a computing node, a rule table corresponding to the request is created based on the information in the original table.

[0022] In step 104, rule table data is generated and sent to a storage node corresponding to the rule table data.

[0023] In step 106, the rule table data is imported into the rule table.

[0024] In step 108, the data in the rule table is supplemented to match the original table.

[0025] In step 110, the data is accessed according to the rules table.

[0026] According to the data reallocation method provided by the first embodiment of the present application, instead of creating a new table to migrate data, a new rule table is created, thereby improving the efficiency of data reallocation and effectively reducing the impact of data migration during reallocation. The fields of the rule table only contain sharding keys and metadata information, which has extremely low storage requirements, reduces overhead, and further improves efficiency.

[0027] EMBODIMENT 2

[0028] Referring to FIG. 2, the data reallocation method according to this embodiment may include the following steps.

[0029] In step 202, a rule table is created.

[0030] When a data redistribution request is submitted, we create a new rule table based on the information of the original table, and the sharding key of the new rule table matches the sharding scheme of the original table of the redistribution request. For example, if the original table is hashed by field col1 and distributed to three nodes, and the redistribution request requires it to be distributed by hash to four nodes, the rule table will be hashed by field col1 and distributed to four nodes.

[0031] In this embodiment, the redistribution that changes the distribution pattern from three nodes to four nodes will be described as an example. The original table name is tbs_info_detail, the distribution pattern is hash(uuid)(g1,g2,g3), and the processor creates a hidden rule table tbs_info_detail_res_rule with the distribution pattern hash(uuid)(g1,g2,g3,g4). The rule table has two fields: uuid and groupid. The attributes of the uuid field are exactly the same as those of the original table, but the attributes of the groupid field can use the unsginedtinyint attribute (occupying only one byte), and the rule requires creating an index on the uuid (if the uuid of the original table is the primary key, set the uuid as the primary key).

[0032] In step 204, data for the rule table is generated.

[0033] Each storage node includes a reallocation control module configured to generate and import metadata rule table data, and these reallocation control modules in parallel export distribution key values ​​from each storage node together with storage node number information, split the split files into files of different storage nodes according to new distribution rules, and transmit them to corresponding storage nodes.

[0034] In this step, all successfully executed INSERT / UPDATE / DELETE statements related to the sharding key are recorded, and the value of the sharding key and the corresponding groupid are recorded.

[0035] In step 206, data is imported into the rules table.

[0036] The reallocation control module of each storage node can import the transmitted data into the rule table.

[0037] In step 208, a read lock is acquired.

[0038] Informs the compute nodes to take read locks on tables that need to be redistributed.

[0039] In step 210, the data is imputed.

[0040] The value of the sharding key recorded in the INSERT / UPDATE / DELETE statement related to the recorded sharding key is compared with the sharding keys in the original table and the rule table, and if data corresponding to the sharding key value is missing in the rule table, data corresponding to the sharding key value in the original table is inserted, if there is extra data corresponding to the sharding key value in the rule table, the data corresponding to the sharding key value is deleted, and if the rule table data does not match the original table, it is corrected to match the original table.

[0041] In step 212, the access method is changed.

[0042] The computing node is notified of the change in the access method of the table, and the computing node's own computation distribution algorithm is changed to access according to the method of the rule table, and INSERT / UPDATE / DELETE statements related to the table need to change the rule table at the same time, and new data is preferentially inserted into the new data node.

[0043] In step 214, the read lock is released.

[0044] Release a previously acquired read lock.

[0045] The redistribution task is now complete.

[0046] In step 216, the distribution rules are aligned.

[0047] Regarding the problem of the rule table distribution method not matching the computing nodes, batch processing can be performed during idle time, for example, as follows.

[0048] 1. First, search for rows in node 1 where the metadata field of the rule table is 2, and then divide that part of the data into batches and migrate them to node 2, while at the same time modifying the rule table.

[0049] 2. Once most of the data has been migrated successfully, lock the table and migrate the remaining data.

[0050] 3. Notify the compute node to change the access method for the table.

[0051] 4. Clean up the rules table.

[0052] According to the data reallocation method provided by embodiment 2 of the present application, a rule table containing only two fields is created, thereby improving the efficiency of reallocation and reducing the impact of data migration while further reducing the overhead of reallocation.

[0053] EMBODIMENT 3

[0054] Referring to FIG. 3, the data redistribution device 300 according to the present embodiment includes:

[0055] The system includes a computing node 301, a processor 302, and a storage node 303. The processor 302 is configured to create a rule table corresponding to the request based on information of the original table in response to a data reallocation request transmitted from the computing node 301. There are a plurality of storage nodes 303, and the system is configured to generate rule table data, transmit the rule table data to the storage node 303 corresponding to the rule table data, import the rule table data into the rule table, and complement the rule table data to match the original table. The computing node 301 is configured to access data according to the method of the rule table.

[0056] According to the data redistribution device provided by embodiment 3 of the present application, instead of creating a new table to migrate data, a rule table is created, thereby improving the efficiency of data redistribution and reducing the impact of data migration.

[0057] EMBODIMENT 4

[0058] Referring to FIG. 4, the data redistribution device 400 according to the present embodiment includes:

[0059] The system includes a calculation node 401, a processor 402, and a storage node 403. There are four storage nodes 403.

[0060] When a data reallocation request is sent from the computing node 401, the processor 402 creates a new rule table based on the information of the original table, and the sharding key of the new rule table matches the sharding scheme of the original table of the reallocation request.

[0061] Each of the four storage nodes 403 includes a reallocation control module 403a configured to generate and import metadata rule table data. These reallocation control modules 403a in parallel export the distribution key value from each storage node 403 together with the storage node number information, split the split files into files of different storage nodes 403 according to the new distribution rules, and transmit them to the corresponding storage nodes 403. The computing node 401 records all the INSERT / UPDATE / DELETE statements related to the sharding key and successfully executed, and records the value of the sharding key and the corresponding groupid.

[0062] The reallocation control module 403a of the storage node 403 imports the transmitted data into the rule table.

[0063] The processor 402 notifies the compute node 401 to place a read lock on the table that needs to be reallocated.

[0064] The reallocation control module 403a compares the value of the sharding key recorded in the INSERT / UPDATE / DELETE statement related to the sharding key recorded in the computation node 401 with the sharding keys in the original table and the rule table, and if data corresponding to the sharding key value is missing from the rule table, inserts data corresponding to the sharding key value in the original table, and if there is extra data corresponding to the sharding key value in the rule table, deletes the data corresponding to the sharding key value, and if the rule table data does not match the original table, modifies it to data that matches the original table.

[0065] The processor 402 notifies the calculation node 401 of the change in the access method of the table, changes the calculation distribution algorithm of the calculation node 401 itself to access according to the method of the rule table, and INSERT / UPDATE / DELETE statements related to the table need to change the rule table at the same time, and new data is preferentially inserted into the new data node.

[0066] The compute node 401 releases the read lock it had acquired, and the redistribution task is now complete.

[0067] If rules need to be distributed to it, then it is performed according to step 216 of embodiment 2.

[0068] According to the data reallocation device provided by embodiment 4 of the present application, by creating a rule table including only two fields, the efficiency of reallocation is improved, the impact of data migration is reduced, and at the same time, the overhead of reallocation is further reduced.

[0069] The present application further provides a data reallocation system including the data reallocation device of the fourth embodiment, which makes it possible to efficiently execute a data reallocation task.

[0070] As can be seen, those skilled in the art will appreciate that all or part of the functional modules / units in the steps of the methods, systems, and devices disclosed above may be implemented as software (which may be implemented by a computer program code executable by a computing device), firmware, hardware, and appropriate combinations thereof. In hardware embodiments, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical assemblies, for example, one physical assembly may have multiple functions, or one function or step may be performed by several physical assemblies in cooperation. Some or all of the physical assemblies may be implemented as software executed by a processor such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as integrated circuits such as application specific integrated circuits.

[0071] The embodiment of the present application further provides an apparatus, and Fig. 5 is a schematic diagram of the configuration of the apparatus provided by the embodiment of the present application. As shown in Fig. 5, the apparatus includes a processor 51, a memory 52, an input device 53, an output device 54, and a communication device 55. The number of processors 51 in the apparatus may be one or more, and Fig. 5 shows one processor 51 as an example. The processor 51, the memory 52, the input device 53, and the output device 54 in the apparatus may be connected by a bus or other method, and Fig. 5 shows an example in which they are connected by a bus.

[0072] The memory 52 can be configured as a computer-readable storage medium to store software programs, computer-executable programs and modules, such as program instructions / modules corresponding to the data reallocation method in the embodiments of the present application. The processor 51 executes the software programs, instructions and modules stored in the memory 52 to perform various functional applications and data processing of the device, i.e., to realize any one of the methods provided by the embodiments of the present application.

[0073] The memory 52 may mainly include a program storage area and a data storage area, where the program storage area may store an operating system, an application program required for at least one function, and the data storage area may store data generated by the use of the device. In addition, the memory 52 may include a high-speed random access memory, or may include a non-volatile memory, such as at least one magnetic disk memory device, a flash memory device, or other non-volatile solid-state memory device. In some examples, the memory 52 may further include a memory located remotely to the processor 51, and these remote memories may be connected to the device via a network. Examples of the above network include, but are not limited to, the Internet, a company intranet, a local area network, a mobile communication network, and combinations thereof.

[0074] The input device 53 can be used to receive entered numeric or textual information and to generate key signal inputs for user settings and function control of the device. The output device 54 can include a display device such as a display.

[0075] The communication device 55 may include a receiver and a transmitter. The communication device 55 is configured to transmit and receive information under the control of the processor 51.

[0076] In one embodiment, an embodiment of the present application further provides a storage medium including computer-executable instructions, which, when executed by a computer processor, are configured to perform a data reallocation method, the data reallocation method including the steps of obtaining a stored procedure of a source database, parsing and translating the stored procedure to obtain a corresponding syntax block list, and processing the syntax block list to obtain a stored procedure that meets requirements of a target database.

[0077] Of course, according to the storage medium containing the computer-executable instructions provided by the embodiments of the present application, the computer-executable instructions are not limited to the operations of the method as described above, but can also perform related operations in the data reallocation method provided by any embodiment of the present application.

[0078] Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be realized by the method of software and necessary general-purpose hardware, which is a better embodiment in many cases, although it can be realized by hardware. Based on this understanding, the essential part of the technical solution of the present application or the part that contributes to the prior art can be embodied in the form of a software product, and this software product can be stored in a computer-readable storage medium, such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk, or an optical disk, which includes some instructions configured to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute the data redistribution method of each embodiment of the present application.

[0079] The above is a detailed description of the embodiments of the present application in conjunction with specific embodiments, and should not be construed as limiting the specific implementation of the present application. Those skilled in the art to which the present application pertains may make some simple deductions or substitutions without departing from the inventive concept of the present application, and all of them should be considered as falling within the scope of protection of the present application.

Claims

1. A data reallocation method using a data reallocation device having a function as a computer, comprising: In response to a data reallocation request transmitted from a computing node, the data reallocation device creates a rule table corresponding to the data reallocation request based on information of an original table; The data redistribution device generates rule table data and transmits the rule table data to a storage node corresponding to the rule table data; said data redistribution device importing said rule table data into said rule table; The data redistribution device supplements the data in the rule table to match the original table; said data redistribution device accessing data according to the rule table; Including, The rule table includes a sharding key and metadata information; The step of generating data for the rule table by the data redistribution device includes: The data redistribution device exports distribution key values ​​and node number information, and divides them into files of corresponding storage nodes according to distribution rules included in the data redistribution request; The data redistribution device records the sharding key and distribution pattern, and the INSERT / UPDATE / DELETE statements successfully executed with the corresponding sharding key; A data reallocation method including:

2. The step of the data redistribution device supplementing the data of the rule table to match the original table comprises the steps of: the data redistribution device comparing the sharding keys of the rule table and the original table with the sharding key recorded in the INSERT / UPDATE / DELETE statement; if data corresponding to the sharding key is missing in the rule table, inserting data corresponding to the sharding key in the original table; if there is excess data corresponding to the sharding key in the rule table, deleting the data corresponding to the sharding key; and if the data corresponding to the sharding key in the rule table does not match the corresponding data in the original table, correcting it to the data corresponding to the sharding key in the original table. The data redistribution method of claim 1 , comprising:

3. Before the step of the data redistribution device supplementing the data of the rule table to match the original table, The data redistribution device further includes a step of placing a read lock on a table that requires data redistribution; After the data redistribution device accesses data according to the rule table, The data redistribution device further includes a step of releasing the read lock. The data reallocation method according to claim 1 or 2.

4. A data reallocation device, a processor configured to, in response to a data reallocation request sent from a computing node, create a rule table corresponding to the data reallocation request based on information in the original table; a storage node configured to generate rule table data, send the rule table data to a storage node corresponding to the rule table data, import the rule table data into the rule table, and complement the rule table data to the original table; a computing node configured to access data according to the rules table; Equipped with The rule table includes a sharding key and metadata information; The storage nodes are further configured to export distribution key values ​​and node number information, and split the information into files of the corresponding storage nodes according to a distribution rule included in the data redistribution request; The computing node is further configured to record the sharding key and distribution pattern, and the INSERT / UPDATE / DELETE statements successfully executed with the corresponding sharding key. Data redistribution device.

5. The storage node is further configured to compare the sharding keys of the rule table and the original table with the sharding keys recorded in the INSERT / UPDATE / DELETE statements, and if data corresponding to the sharding key is missing from the rule table, to insert data corresponding to the sharding key in the original table, if there is excess data corresponding to the sharding key in the rule table, to delete the data corresponding to the sharding key, and if there is a mismatch between the data corresponding to the sharding key in the rule table and the corresponding data in the original table, to modify the data to the data corresponding to the sharding key in the original table. The data redistribution device according to claim 4.

6. The computing node is further configured to: before the storage node complements the data, place a read lock on a table requiring data redistribution; and after accessing the data according to the method of the rule table, release the read lock.

6. The data redistribution device according to claim 4 or 5.

7. A data redistribution device according to any one of claims 4 to 6. Data redistribution system.

8. An apparatus comprising: one or more processors; a memory in which one or more programs are stored; Equipped with The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method according to any one of claims 1 to 3. device.

9. A storage medium storing a computer program, The computer program, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 3. storage medium.

Images