Data table processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data table processing method and device, equipment, a storage medium and a program product, and relates to the field of big data. The method comprises the steps of obtaining at least one partition of a target data table; determining at least one target fragment; sorting the partition data blocks in the partitions according to the record line number of the partition data blocks to obtain a to-be-allocated list corresponding to the partitions; sorting the target fragments according to the storage information of the target fragments to obtain a fragment sequence; generating a balanced task list according to the list to be allocated and the fragment sequence; and distributing the target data block to the target fragment corresponding to the target data block. According to the method, the data balancing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a data table processing method, device, equipment, storage medium and program product. Background Art

[0002] Based on the distributed database service, for the storage of big data, it is necessary to balance the data in the data table, that is, split the data table and perform distributed storage processing.

[0003] In the prior art, data redistribution is performed on the data in the data table to be processed based on a hash consistency algorithm.

[0004] However, in the above method, it is necessary to scan and calculate each data in the data table to be processed, which is a complicated and time-consuming process, resulting in reduced data table processing efficiency and further reduced data balancing efficiency. Summary of the invention

[0005] The present application provides a data table processing method, device, equipment, storage medium and program product to solve the technical problem that a data redistribution processing method based on a hash consistency algorithm reduces data balancing efficiency.

[0006] In a first aspect, the present application provides a method for processing a data table, the method comprising:

[0007] Acquire at least one partition of the target data table; and determine at least one target shard; wherein the partition includes at least one partition data block, and the partition data block is used to store record data in the partition; the target shard is used to store and process data in the target data table;

[0008] According to the number of record rows of the partition data block, the partition data blocks in each partition are sorted to obtain a to-be-allocated list corresponding to each partition; wherein the number of record rows is the number of rows of record data stored in the partition data block;

[0009] According to the storage information of each target shard, each target shard is sorted to obtain a shard sequence; wherein the storage information represents the storage space occupied by the partition in the target shard;

[0010] Generate a balancing task list according to the to-be-allocated list and the shard sequence; wherein the balancing task list includes at least one target data block and a target shard corresponding to the target data block; the target data block is a partition data block to be migrated out of the to-be-allocated list;

[0011] Allocate the target data block to the target slice corresponding to the target data block.

[0012] In a possible implementation, generating a balancing task list according to the to-be-allocated list and the shard sequence includes:

[0013] If it is determined that the number of data blocks in the partition is greater than or equal to the number of shards, the balancing task list is generated according to the list to be allocated and the shard sequence; wherein the number of data blocks is the number of data blocks in each partition in the partition; and the number of shards is the number of each target shard.

[0014] In a possible implementation, generating the balancing task list according to the to-be-allocated list and the shard sequence includes:

[0015] For each partition data block in the to-be-allocated list, determine an average value corresponding to the partition where the partition data block is located according to the number of data blocks of the partition where the partition data block is located and the number of shards;

[0016] If it is determined that the number of data blocks corresponding to the partition data block is greater than the average value corresponding to the partition data block, then determining the partition data block as the target data block;

[0017] The target shard corresponding to the target data block is determined from the shard sequence, and the number of data blocks of the partition where the target data block is located is updated.

[0018] In a possible implementation, allocating the target data block to a target slice corresponding to the target data block includes:

[0019] If it is determined that the target data table is not in a data table change operation state, then closing the data merging process for the target data table; and renaming the directory name of the target data block;

[0020] Copying the target data block according to the migration-out node and the migration-in node of the target data block; wherein the migration-out node is the node where the target data block is located before the copy processing; and the migration-in node is the node where the target data block is located after the copy processing;

[0021] Based on the migration node, restoring the directory name of the target data block;

[0022] Verify the file information of the target data block and the file information of the copied target data block; wherein the file information includes a directory name and a file size.

[0023] In a possible implementation manner, after restoring the directory name of the target data block, the method further includes:

[0024] In response to the task stop instruction, stopping the data balancing process on the target data table;

[0025] Performing rollback processing on each of the partition data blocks;

[0026] The copied target data block in the migration node is deleted.

[0027] In a possible implementation, the method further includes:

[0028] Unloading the structured language corresponding to the target data block in the migration-out node;

[0029] Execute a directory name switching command of the target data block in the migration-out node;

[0030] Loading the structured language corresponding to the copied target data block in the migration node;

[0031] Execute the data table statement in the migration node; or execute the data table statement in the migration node; wherein the data table statement represents the start of data merging processing on the target data table.

[0032] In a possible implementation, the method further includes:

[0033] If it is determined that the first data volume and the second data volume of the target data table are consistent, the target data block that has been unloaded and processed in the migrated node is deleted based on the migrated node; wherein the first data volume is the table data volume of the target data table before the balanced processing; the second data volume is the table data volume of the target data table after the balanced processing.

[0034] In a second aspect, the present application provides a data table processing device, the device comprising:

[0035] A determination module is used to obtain at least one partition of the target data table; and determine at least one target shard; wherein the partition includes at least one partition data block, and the partition data block is used to store the record data in the partition; the target shard is used to store and process the data in the target data table;

[0036] A first sorting module is used to sort the partition data blocks in each partition according to the number of record rows of the partition data block to obtain a to-be-allocated list corresponding to each partition; wherein the number of record rows is the number of rows of record data stored in the partition data block;

[0037] A second sorting module is used to sort each of the target shards according to the storage information of each of the target shards to obtain a shard sequence; wherein the storage information represents the storage space occupied by the partition in the target shard;

[0038] A generating module, configured to generate a balancing task list according to the to-be-allocated list and the shard sequence; wherein the balancing task list includes at least one target data block and a target shard corresponding to the target data block; the target data block is a partition data block to be migrated out of the to-be-allocated list;

[0039] The allocation module is used to allocate the target data block to the target slice corresponding to the target data block.

[0040] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0041] The memory stores computer-executable instructions;

[0042] The processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect.

[0043] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0044] In a fifth aspect, the present application provides a computer program product, the computer program product comprising: a computer program, and when the computer program is executed by a processor, it is used to implement the method described in the first aspect.

[0045] The data table processing method, device, equipment, storage medium and program product provided by the present application respectively sort the partition data blocks and target shards of the target data table to obtain a list to be allocated and a shard sequence, and generate a balancing task list based on the list to be allocated and the shard sequence, and according to the balancing list, each target data block of the target data table can be allocated to the target shard corresponding to the target data block; thereby, the data balancing efficiency of the data table is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] Figure 1 A schematic diagram of an application scenario provided for this application;

[0048] Figure 2 A flowchart of a method for processing a data table provided in an embodiment of the present application;

[0049] Figure 3 A flowchart of another data table processing method provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of a reduced capacity delivery process provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of a capacity expansion delivery process provided in an embodiment of the present application;

[0052] Figure 6 A flowchart of another method for processing a data table provided in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of the structure of a data table processing device provided in an embodiment of the present application;

[0054] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0055] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0056] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0057] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0058] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0059] First, the terms involved in this application are explained:

[0060] Shard: Store massive amounts of data in multiple nodes in a distributed manner. Each node only stores and processes a portion of the massive data. When there is a single copy, one shard corresponds to one node. When there are multiple copies, one shard corresponds to multiple nodes.

[0061] Replica: In order to ensure data security and high availability of services under abnormal circumstances, data is redundantly stored in multiple nodes, and the data of multiple nodes are copies of each other.

[0062] Partition: In a column-based database management system, a partition is a virtual instance with no physical representation. The partition is specified based on the keywords that are used when the business creates the table.

[0063] Based on distributed database services, the storage of big data requires data balancing, that is, splitting the data and performing distributed storage processing.

[0064] In one example, based on a hash consistency algorithm, data redistribution is performed on the data in the data table to be processed.

[0065] However, in the above method, it is necessary to scan and calculate each data in the data table to be processed, which is a complicated and time-consuming process, resulting in reduced data table processing efficiency and further reduced data balancing efficiency.

[0066] In one example, data redistribution is achieved by rewriting the original table data evenly to each shard according to the sharding rules of the distributed table; for example, on each shard, loop through the data table under the database, create a temporary table based on the data table, create a temporary distributed table based on the temporary table, and write the data of the data table evenly to the temporary table of each shard according to the business rules of the temporary distributed table. The data redistribution statement can be executed in parallel on each shard. After the data is confirmed to be correct, swap the data table name and the temporary table name, and delete the temporary table and the temporary distributed table.

[0067] However, in the above method, it is necessary to create a temporary table and then rewrite the data into the temporary table. The temporary data in the process requires additional storage space, which is prone to insufficient disk space and causes failure of data balancing tasks. In addition, due to the small number of concurrencies, the time cost of data redistribution is very high and the system change cycle is long, which affects the efficiency of data balancing.

[0068] In response to the above technical problems, the present application proposes the following technical concept: by sorting each partition data block and each target shard of the target data table, a list to be allocated and a shard sequence are obtained, and a balancing task list is generated according to the list to be allocated and the shard sequence, and according to the balancing list, each target data block of the target data table can be allocated to the target shard corresponding to the target data block. In this way, the problem that the data redistribution processing method based on the hash consistency algorithm in the prior art reduces the efficiency of data balancing can be solved.

[0069] Figure 1A schematic diagram of an application scenario provided by this application. Figure 1 As shown, the specific application scenario of the present application includes an electronic device 101 and a user 102. The user 102 can perform data balancing processing on one or more current data tables through the electronic device 101.

[0070] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0071] Figure 2 A flowchart of a method for processing a data table provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes:

[0072] 201. Obtain at least one partition of a target data table; and determine at least one target shard; wherein the partition includes at least one partition data block, and the partition data block is used to store record data in the partition; the target shard is used to store and process data in the target data table.

[0073] Exemplarily, the execution subject of this embodiment may be an electronic device, hereinafter referred to as the device. The device is a physical device or a virtual device that can execute a data table processing method. When the user needs to perform data balancing processing, or the device automatically performs data balancing based on a scheduled task, the device can obtain one or more data tables and data table information in a distributed system by executing a specified structured query language (SQL) statement. Due to the multiple copies used in the actual production cluster, system tables, views, external tables and other table types that do not need to be redistributed are excluded. For each data table, the device determines the specified data table, that is, the target data table, and obtains all partitions of each target data table, wherein each partition includes multiple partition data blocks (Part), and each partition data block is used to store the record data in the partition; at the same time, the device determines multiple target shards for storing and processing part of the data of the target data table.

[0074] 202. Sort the partition data blocks in each partition according to the number of record rows of the partition data blocks to obtain a to-be-allocated list corresponding to the partition; wherein the number of record rows is the number of rows of record data stored in the partition data blocks.

[0075] Exemplarily, for each partition data block, each partition data block has a record row number, that is, the number of rows of record data stored in the partition data block. The device sorts the partition data blocks in all partitions according to the record row number of each partition data block, such as sorting them from large to small by the number of Part record rows, and storing them in the PART list to be allocated, so as to obtain the list to be allocated corresponding to each partition.

[0076] 203. Sort the target shards according to the storage information of the target shards to obtain a shard sequence; wherein the storage information represents the storage space occupied by the partitions in the target shards.

[0077] Exemplarily, for each target shard, each target shard has storage information, that is, the storage space occupied by the partitions in the target shard. The device sorts all target shards according to the storage information of each target shard, such as sorting them from small to large by storage space, and stores them in a shard sequence.

[0078] 204. Generate a balancing task list according to the list to be allocated and the shard sequence; wherein the balancing task list includes at least one target data block and a target shard corresponding to the target data block; wherein the target data block is a partition data block to be migrated out in the list to be allocated.

[0079] Exemplarily, in order to execute the balancing task, the device needs to generate the balancing task first, the purpose of which is to determine which PART file of which partition the migrating node migrates to the specified directory of the migrating node, that is, according to the obtained list to be allocated and the shard sequence, determine the partition data blocks to be migrated in the list to be allocated. If the list to be allocated is sorted in the order of the number of record rows, the first few in the sorting are the target data blocks, and according to the arrangement order of the sorted target shards in the shard sequence, the partition data blocks with a large number of record rows can be delivered to the target shard with a small storage space occupied by the partition. Each target data block and the target shard corresponding to each target data block can be determined and stored in the balancing task list, and the balancing task can be generated based on the information in the balancing task list.

[0080] 205. Allocate the target data block to the target shard corresponding to the target data block.

[0081] Exemplarily, the device allocates each target data block in the balancing task list to the target shard corresponding to the target data block according to the generated balancing task list, and the target shard can perform further storage and processing, thereby completing the balancing process of the data table.

[0082] In this embodiment, a data table processing method is provided, by sorting each partition data block and each target shard of the target data table respectively, obtaining a list to be allocated and a shard sequence, generating a balancing task list according to the list to be allocated and the shard sequence, and according to the balancing list, each target data block of the target data table can be allocated to the target shard corresponding to the target data block; further, data balancing is performed using three-level granularity of shard, partition, and Part, so the algorithm can also be applied to the scenario of only wanting to redistribute part of the partitions of a table. Compared with the two-level granularity based on shard and partition, the three-level granularity can disassemble more task units, which is more conducive to the parallel execution of multiple tasks, thereby greatly shortening the time for data balancing. At the same time, based on the three-level granularity, the data after data redistribution can be made more balanced, thereby improving the data table processing efficiency and data balancing efficiency.

[0083] Figure 3 A flowchart of another method for processing a data table provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the method includes:

[0084] 301. Obtain at least one partition of a target data table; and determine at least one target shard; wherein the partition includes at least one partition data block, and the partition data block is used to store record data in the partition; the target shard is used to store and process data in the target data table.

[0085] Exemplarily, this step may refer to step 201 and will not be described in detail here.

[0086] 302. Sort the partition data blocks in each partition according to the number of record rows of the partition data blocks to obtain a to-be-allocated list corresponding to each partition; wherein the number of record rows is the number of rows of record data stored in the partition data blocks.

[0087] Exemplarily, this step may refer to step 202 and will not be described in detail here.

[0088] 303. Sort the target shards according to the storage information of the target shards to obtain a shard sequence; wherein the storage information represents the storage space occupied by the partitions in the target shards.

[0089] Exemplarily, this step may refer to step 203 and will not be described in detail here.

[0090] 304. If it is determined that the number of data blocks of the partition is greater than or equal to the number of shards, a balancing task list is generated according to the to-be-allocated list and the shard sequence corresponding to the partition; wherein the number of data blocks is the number of data blocks of each partition in the partition; and the number of shards is the number of each target shard.

[0091] Exemplarily, based on the principle of average distribution, the device needs to first determine the number of data blocks in each partition, that is, the number of data blocks, and determine the corresponding number of each target shard, that is, the number of shards. For each partition, the number of data blocks in each partition is compared with the number of shards. If the number of data blocks in the partition is less than the number of shards, it means that the partition does not need data and exits directly. If the number of data blocks in the partition is greater than or equal to the number of shards, each list to be allocated can be looped through and delivered to the shard sequence from top to bottom in sequence. When a round of delivery is completed, the number of partition data blocks in each partition is recalculated, and then delivered one by one from top to bottom in sequence until the number of remaining partition data blocks in the list to be allocated is less than the number of shards. The partition data blocks of the target shards that have been determined to be delivered in each list to be allocated are determined as the target data blocks in the list to be allocated, and a balanced task list can be obtained. Figure 4 A schematic diagram of a reduced capacity delivery process provided in an embodiment of the present application, such as Figure 4 As shown, during the shrinking delivery process, the target data blocks in the to-be-allocated list are arranged from large to small according to the number of record rows, and the target shards in the shard sequence are arranged from small to large according to the storage space corresponding to the storage information. Based on the principle of average distribution, data balancing is performed.

[0092] In one example, step 304 generates a balancing task list according to the to-be-allocated list and the shard sequence, including the following steps:

[0093] The first step of step 304 is to determine, for each partition data block in the to-be-allocated list, an average value corresponding to the partition where the partition data block is located according to the number of data blocks and the number of shards in the partition where the partition data block is located.

[0094] In the second step of step 304, if it is determined that the number of data blocks corresponding to the partition data block is greater than the average value corresponding to the partition data block, the partition data block is determined to be the target data block.

[0095] The third step of step 304 is to determine the target shard corresponding to the target data block from the shard sequence, and update the number of data blocks of the partition where the target data block is located.

[0096] Exemplarily, when determining whether each partition data block in the to-be-allocated list corresponding to each partition needs to be delivered, based on the minimum movement principle, for each partition data block in the to-be-allocated list, the device first divides the number of data blocks in the partition where the partition data block is located by the number of shards corresponding to each target shard, to obtain the corresponding average value of the partition data block, and then compares the number of data blocks corresponding to the partition data block with the average value corresponding to the partition data block. If the number of data blocks corresponding to the partition data block is greater than the average value corresponding to the partition data block, it is determined that the partition data block can be delivered and moved out, that is, the partition data block is the target data block, and according to the order of the target data block in the to-be-allocated list, the target shard in the shard sequence that is consistent with the order is determined as the target shard corresponding to the target data block; Figure 5 A schematic diagram of a capacity expansion delivery process provided in an embodiment of the present application, such as Figure 5 As shown, in the process of capacity expansion and delivery, the target data blocks in the list to be allocated are arranged from large to small according to the number of record rows, and the target shards in the shard sequence are arranged from small to large according to the storage space corresponding to the storage information. Based on the principle of average distribution and the principle of minimum movement, data balancing is performed, that is, the first target data block in the list to be allocated corresponds to the first target shard in the shard sequence, and the i-th target data block in the list to be allocated corresponds to the i-th target shard in the shard sequence, where i is an integer greater than or equal to 1; after a target data block is determined, it is necessary to re-update the number of data blocks in the partition where the target data block is located, that is, the number of data blocks corresponding to the target data block is reduced by 1, and at the same time, the allocation pointer in the shard sequence is moved downward to determine the next target shard of the target shard corresponding to the target data block, which is the target shard corresponding to the next target data block. It is worth noting that if the number of data blocks corresponding to the i-th partition data block is less than or equal to the average value corresponding to the i-th partition data block, the i-th partition data block is removed from the list to be allocated. Furthermore, based on the Part number average distribution algorithm, the disk usage capacity is taken into account, and the minimum movement principle is set to reduce invalid data copying.

[0097] 305. Allocate the target data block to the target shard corresponding to the target data block.

[0098] Exemplarily, this step may refer to step 205 and will not be described in detail here.

[0099] In this embodiment, on the basis of the above embodiment, based on the rule of average distribution of the number of Parts and taking into account the disk usage capacity, a minimum movement constraint is set in the delivery distribution. On the one hand, it reduces the generation of invalid tasks, thereby reducing invalid data copying; on the other hand, the disk usage capacity is based on the number of records in the specified table and partition, and the correctness of data distribution will not be interfered with by non-business data such as system logs or software installation media.

[0100] Figure 6 A flowchart of another method for processing a data table provided in an embodiment of the present application is shown as follows: Figure 6 As shown, the method includes:

[0101] 401. Obtain at least one partition of a target data table; and determine at least one target shard; wherein the partition includes at least one partition data block, and the partition data block is used to store record data in the partition; the target shard is used to store and process data in the target data table.

[0102] Exemplarily, this step may refer to step 201 and will not be described in detail here.

[0103] 402. Sort the partition data blocks in each partition according to the number of record rows of the partition data blocks to obtain a to-be-allocated list corresponding to each partition; wherein the number of record rows is the number of rows of record data stored in the partition data blocks.

[0104] Exemplarily, this step may refer to step 202 and will not be described in detail here.

[0105] 403. Sort the target shards according to the storage information of the target shards to obtain a shard sequence; wherein the storage information represents the storage space occupied by the partitions in the target shards.

[0106] Exemplarily, this step may refer to step 203 and will not be described in detail here.

[0107] 404. Generate a balancing task list according to the list to be allocated and the shard sequence; wherein the balancing task list includes at least one target data block and a target shard corresponding to the target data block; wherein the target data block is a partition data block to be migrated out in the list to be allocated.

[0108] Exemplarily, this step may refer to step 204 and will not be described in detail here.

[0109] 405. If it is determined that the target data table is not in a data table change operation state, then close the data merge process for the target data table; and rename the directory name of the target data block.

[0110] Exemplarily, the device loops through the generated balancing task list, executes the data copy task, and first determines whether the target data table is in the data table change operation state. If it is determined that the target data table is in the data table change operation state, it is necessary to wait until the target data table is not in the data table change operation state; if it is determined that the target data table is not in the data table change operation state, then the data merge processing of the target data table is closed to ensure that the table partition data blocks are no longer merged, and the same library table can be executed only once. Since there is a risk of triggering a merge during the DETACH / ATTACH operation, it is necessary to rename the migrated Part file, that is, each target data block, to avoid the migrated Part file from being merged. Even if the partition where the Part is located is merged, the renamed Part will not be merged in. At this time, the Part metadata file still exists; if there are multiple copies, the renaming needs to be performed on multiple copies at the same time, otherwise the synchronization mechanism between copies will make the result less than expected.

[0111] 406. Copy the target data block according to the migration-out node and the migration-in node of the target data block; wherein the migration-out node is the node where the target data block is located before the copy processing; and the migration-in node is the node where the target data block is located after the copy processing.

[0112] Exemplarily, the device generates a balancing task record based on each target data block, in which the format is: library name, table name, partition name, Part name, migration node, migration node, the migration node is the node where the target data block is located before being copied, and the migration node is the node where the target data block is located after being copied. The device can copy each target data block according to the migration node and migration node of each target data block; for example, the rsync command is used on the migration node to copy the data of the target data block to the detached directory of the migration node, and the original directory name of the copied target data block is restored at the same time.

[0113] 407. Based on the migration node, restore the directory name of the target data block.

[0114] Exemplarily, after the device completes copying the target data block to the migration node, on the migration node, the directory name of each target data block subjected to data copy processing is restored to the original directory name.

[0115] In one example, after the directory name of the target data block is restored, the following steps are also included:

[0116] Step 1: In response to a task stop instruction, stop performing data balancing processing on the target data table.

[0117] Step 2: Roll back the data blocks of each partition.

[0118] Step 3: Delete the copied target data block in the migration node.

[0119] Exemplarily, when a data balancing task is being executed, the data balancing task can be canceled at any time. If the user needs to stop the data balancing processing, a task stop instruction is input to the device. The device responds to the task stop instruction and stops performing data balancing processing on the target data table. Then, each partition data block, including processed and unprocessed partition data blocks, is rolled back. For example, the name of the rollback target data block is determined, and its corresponding multiple copies all need to perform this operation. Then, the migration node corresponding to each target data block is determined, and based on each migration node, the copied target data block in the migration node is deleted, such as deleting the copied target data block under the detach directory on the migration node server.

[0120] 408. Verify the file information of the target data block and the file information of the copied target data block; wherein the file information includes a directory name and a file size.

[0121] Exemplarily, the device obtains the file information of the target data block and the file information of the corresponding copied target data block, which includes the directory name and file size, and verifies whether the directory name and file size of the migrated-out file, i.e., the target data block, and the migrated-in file, i.e., the copied target data block are consistent, so as to ensure the accuracy of data balancing.

[0122] In one example, it also includes:

[0123] Step 1: Unload the structured language corresponding to the target data block in the migrated node.

[0124] Step 2: Execute a directory name switching command of the target data block in the migration node.

[0125] Step 3: Load the structured language corresponding to the copied target data block in the migration node.

[0126] Step 4: Execute the data table statement in the migration node; or, execute the data table statement in the migration node; wherein the data table statement indicates starting the data merging process of the target data table.

[0127] Exemplarily, after obtaining the balancing task list, the device loops through the balancing task list and executes the data switching process, first unloading the structured language corresponding to the target data block in each outgoing node, and according to the aforementioned directory renaming operation, executing the directory name switching command of the target data block in the specified directory in each outgoing node, and then loading the structured language corresponding to the copied target data block in each incoming node, and executing the data table statement in each incoming node; or, executing the data table statement in each outgoing node to start data merging processing of the target data table.

[0128] 409. If it is determined that the first data volume and the second data volume of the target data table are consistent, based on the migration node, delete the target data block that has been unloaded and processed in the migration node; wherein the first data volume is the table data volume of the target data table before the balancing process; the second data volume is the table data volume of the target data table after the balancing process.

[0129] Exemplarily, after executing the data balancing task, the device needs to clean up the current target data table, so the table data volume of the current target data table before balancing processing, that is, the first data volume, is determined, and the table data volume of the target data table after balancing processing, that is, the second data volume, can be compared to see if the two are consistent. If it is determined that the two are consistent, a drop detached operation is performed at the preset data instance on the migration node, that is, the node where the target data table is located, so that the data of the target partition blocks in each partition of the target data table can be deleted, and the unloaded target data blocks in the migration node can be deleted; furthermore, only the network and disk throughput of the migration node are involved, and a larger concurrent copy can be achieved, thereby reducing the time cost and hardware cost of performing data balancing after the cluster scale changes.

[0130] In this embodiment, based on the above embodiment, a renaming process is added when executing the uninstallation of the original target data block. The original directory is first renamed in all copies of the shard, and the original directory name is restored when copied to the target shard, so that the target data block directory under the uninstallation of the original target data block is not merged, ensuring the complete implementation of the process.

[0131] Figure 7 A schematic diagram of a data table processing device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the device comprises:

[0132] Determination module 501 is used to obtain at least one partition of the target data table; and determine at least one target shard; wherein the partition includes at least one partition data block, and the partition data block is used to store the record data in the partition; the target shard is used to store and process the data in the target data table.

[0133] The first sorting module 502 is used to sort the partition data blocks in each partition according to the number of record rows of the partition data blocks to obtain a to-be-allocated list corresponding to each partition; wherein the number of record rows is the number of rows of record data stored in the partition data block;

[0134] The second sorting module 503 is used to sort each target shard according to the storage information of each target shard to obtain a shard sequence; wherein the storage information represents the storage space occupied by the partitions in the target shard.

[0135] The generation module 504 is used to generate a balancing task list according to the list to be allocated and the shard sequence; wherein the balancing task list includes at least one target data block and a target shard corresponding to the target data block; the target data block is a partition data block to be migrated out in the list to be allocated.

[0136] The allocation module 505 is used to allocate the target data block to the target slice corresponding to the target data block.

[0137] In a possible implementation, the generation module 504 is specifically used to: if it is determined that the number of data blocks of a partition is greater than or equal to the number of shards, then generate a balancing task list according to the to-be-allocated list and shard sequence corresponding to the partition; wherein the number of data blocks is the number of data blocks in each partition in the partition; and the number of shards is the number of each target shard.

[0138] In a possible implementation, the generation module 504 is specifically used to: for each partition data block in the to-be-allocated list, determine the average value corresponding to the partition where the partition data block is located based on the number of data blocks and the number of shards of the partition where the partition data block is located; if it is determined that the number of data blocks corresponding to the partition data block is greater than the average value corresponding to the partition data block, determine that the partition data block is the target data block; determine the target shard corresponding to the target data block from the shard sequence, and update the number of data blocks of the partition where the target data block is located.

[0139] In one possible implementation, the allocation module 505 is specifically used for: if it is determined that the target data table is not in a data table change operation state, closing the data merge processing of the target data table; and renaming the directory name of the target data block; copying the target data block according to the outgoing node and the incoming node of the target data block; wherein the outgoing node is the node where the target data block is located before being copied; the incoming node is the node where the target data block is located after being copied; based on the incoming node, restoring the directory name of the target data block; verifying the file information of the target data block and the file information of the copied target data block; wherein the file information includes the directory name and the file size.

[0140] In a possible implementation, the allocation module 505 is further specifically used to: in response to a task stop instruction, stop data balancing processing on the target data table; perform rollback processing on each partition data block; and delete the copied target data block in the migrated node.

[0141] In a possible implementation, the allocation module 505 is also specifically used to: unload the structured language corresponding to the target data block in the migrated node; execute the directory name switching command of the target data block in the migrated node; load the structured language corresponding to the copied target data block in the migrated node; execute the data table statement in the migrated node; or, execute the data table statement in the migrated node; wherein the data table statement represents the start of data merging processing of the target data table.

[0142] In a possible implementation, the device is also used to: if it is determined that the first data volume and the second data volume of the target data table are consistent, then based on the migration node, delete the target data block that has been unloaded and processed in the migration node; wherein the first data volume is the table data volume of the target data table before the balancing processing; the second data volume is the table data volume of the target data table after the balancing processing.

[0143] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0144] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. Each module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The functions of each module. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0145] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the electronic device includes: a memory 601 and a processor 602; the memory 601 is a memory for storing instructions executable by the processor 602.

[0146] The processor 602 is configured to execute the method provided in the above embodiment.

[0147] The electronic device further includes a receiver 603 and a transmitter 604. The receiver 603 is used to receive instructions and data sent by other devices, and the transmitter 604 is used to send instructions and data to external devices.

[0148] An embodiment of the present application also provides a chip for running instructions, which is used to execute the technical solutions in the above embodiments.

[0149] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed on a computer, the computer executes the technical solution in the above embodiment.

[0150] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, the technical solution in the above embodiments can be implemented.

[0151] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0152] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0153] It should be understood that the above processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0154] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0155] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0156] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0157] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0158] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0159] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for processing a data table, characterized in that: The method comprises: Acquire at least one partition of the target data table; and determine at least one target shard; wherein the partition includes at least one partition data block, and the partition data block is used to store record data in the partition; the target shard is used to store and process data in the target data table; According to the number of record rows of the partition data block, the partition data blocks in each partition are sorted to obtain a to-be-allocated list corresponding to each partition; wherein the number of record rows is the number of rows of record data stored in the partition data block; According to the storage information of each target shard, each target shard is sorted to obtain a shard sequence; wherein the storage information represents the storage space occupied by the partition in the target shard; Generate a balancing task list according to the to-be-allocated list and the shard sequence; wherein the balancing task list includes at least one target data block and a target shard corresponding to the target data block; the target data block is a partition data block to be migrated out of the to-be-allocated list; Allocate the target data block to the target slice corresponding to the target data block.

2. The method according to claim 1, characterized in that: Generate a balancing task list according to the to-be-allocated list and the shard sequence, including: If it is determined that the number of data blocks in the partition is greater than or equal to the number of shards, the balancing task list is generated according to the list to be allocated and the shard sequence; wherein the number of data blocks is the number of data blocks in each partition in the partition; and the number of shards is the number of each target shard.

3. The method according to claim 2, characterized in that Generating the balancing task list according to the to-be-allocated list and the shard sequence includes: For each partition data block in the to-be-allocated list, determine an average value corresponding to the partition where the partition data block is located according to the number of data blocks of the partition where the partition data block is located and the number of shards; If it is determined that the number of data blocks corresponding to the partition data block is greater than the average value corresponding to the partition data block, then determining the partition data block as the target data block; The target shard corresponding to the target data block is determined from the shard sequence, and the number of data blocks of the partition where the target data block is located is updated.

4. The method according to claim 1, characterized in that Allocating the target data block to a target shard corresponding to the target data block includes: If it is determined that the target data table is not in a data table change operation state, then closing the data merging process for the target data table; and renaming the directory name of the target data block; Copying the target data block according to the migration-out node and the migration-in node of the target data block; wherein the migration-out node is the node where the target data block is located before the copy processing; and the migration-in node is the node where the target data block is located after the copy processing; Based on the migration node, restoring the directory name of the target data block; Verify the file information of the target data block and the file information of the copied target data block; wherein the file information includes a directory name and a file size.

5. The method according to claim 4, characterized in that After restoring the directory name of the target data block, the method further includes: In response to the task stop instruction, stopping the data balancing process on the target data table; Performing rollback processing on each of the partition data blocks; The copied target data block in the migration node is deleted.

6. The method according to claim 4, characterized in that The method further comprises: Unloading the structured language corresponding to the target data block in the migration-out node; Execute a directory name switching command of the target data block in the migration-out node; Loading the structured language corresponding to the copied target data block in the migration node; Execute the data table statement in the migration node; or execute the data table statement in the migration node; wherein the data table statement represents the start of data merging processing on the target data table.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: If it is determined that the first data volume and the second data volume of the target data table are consistent, the target data block that has been unloaded and processed in the migrated node is deleted based on the migrated node; wherein the first data volume is the table data volume of the target data table before the balanced processing; the second data volume is the table data volume of the target data table after the balanced processing.

8. A data table processing device, characterized in that: The device comprises: A determination module is used to obtain at least one partition of the target data table; and determine at least one target shard; wherein the partition includes at least one partition data block, and the partition data block is used to store the record data in the partition; the target shard is used to store and process the data in the target data table; A first sorting module is used to sort the partition data blocks in each partition according to the number of record rows of the partition data block to obtain a to-be-allocated list corresponding to each partition; wherein the number of record rows is the number of rows of record data stored in the partition data block; A second sorting module is used to sort each of the target shards according to the storage information of each of the target shards to obtain a shard sequence; wherein the storage information represents the storage space occupied by the partition in the target shard; A generating module, configured to generate a balancing task list according to the to-be-allocated list and the shard sequence; wherein the balancing task list includes at least one target data block and a target shard corresponding to the target data block; the target data block is a partition data block to be migrated out of the to-be-allocated list; The allocation module is used to allocate the target data block to the target slice corresponding to the target data block.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.

Images