Distributed data table migration method and device, equipment and medium

Abstract

The invention discloses a distributed data table migration method and device, equipment and a medium. The method comprises the following steps: cutting source table data to obtain a plurality of sub-files; sampling the sub-files according to a sampling proportion to obtain sampling data; fragmenting the sampled data by adopting a fragmentation rule and a fragmentation key of the target data table to obtain a target fragmentation proportion of the sub-files; determining a target file proportion queue of the sub-files from the candidate file proportion queues according to a target fragment proportion based on the fragment proportion associated with the candidate file proportion queues, and storing the sub-files in the target file proportion queue; according to the candidate file proportion queue, the data capacity of the sub-files, a first migration queue and a second migration queue, selecting the sub-files from the candidate file proportion queue and moving the sub-files to the first migration queue; and storing the sub-files in the first migration queue in a target data table in parallel. According to the embodiment of the invention, the data migration efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data migration, and in particular to a distributed data table migration method, device, equipment and medium. Background Art

[0002] Database migration technology generally refers to the permanent transfer of data from one storage system or database management system to another. This process involves data selection, extraction, conversion, migration, and verification.

[0003] The existing technology has actively explored the database migration performance. During the migration process, the efficiency of the migration process is often improved by monitoring the migration machine and data node machine resources and dynamically adjusting the migration parameter configuration during the migration process, while ignoring the impact of data distribution in the data files to be migrated on the migration efficiency. Summary of the invention

[0004] The present invention provides a distributed data table migration method, device, equipment and medium to solve the impact of data distribution of data files to be migrated on migration efficiency when data is migrated to a distributed database, thereby improving the distributed data table migration efficiency.

[0005] According to one aspect of the present invention, a distributed data table migration method is provided, comprising:

[0006] Cut the source table data into multiple sub-files;

[0007] Sampling the sub-files according to the sampling ratio to obtain sampling data;

[0008] The sampled data is sliced ​​using a slicing rule and a slicing key of a target data table to obtain a target slicing ratio of subfiles; the target data table is stored on at least one data node;

[0009] Based on the shard ratio associated with the candidate file ratio queue, determine the target file ratio queue of the subfile from the candidate file ratio queue according to the target shard ratio, and store the subfile in the target file ratio queue;

[0010] According to the candidate file proportion queue, the data capacity of the sub-file, the first migration queue, and the second migration queue, selecting a sub-file from the candidate file proportion queue and moving the sub-file to the first migration queue;

[0011] The sub-files in the first migration queue are stored in parallel in the target data table.

[0012] According to another aspect of the present invention, there is provided a distributed data table migration device, comprising:

[0013] The source data segmentation module is used to segment the source table data into multiple sub-files;

[0014] A sub-file sharding ratio module is used to sample sub-files according to a sampling ratio to obtain sampled data; slice the sampled data using a sharding rule and a sharding key of a target data table to obtain a target sharding ratio of the sub-files; the target data table is stored on at least one data node;

[0015] A sub-file division module, configured to determine a target file proportion queue of the sub-file from the candidate file proportion queue according to the target shard proportion based on the shard proportion associated with the candidate file proportion queue, and store the sub-file in the target file proportion queue;

[0016] The sub-file migration module is used to select a sub-file from the candidate file ratio queue and move the sub-file to the first migration queue according to the candidate file ratio queue, the data capacity of the sub-file and the first migration queue, and the second migration queue; and store the sub-files in the first migration queue in parallel to the target data table.

[0017] According to another aspect of the present invention, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the distributed data table migration method according to any embodiment of the present invention is implemented.

[0018] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0019] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the distributed data table migration method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the distributed data table migration method described in any embodiment of the present invention when executed.

[0021] The embodiment of the present invention samples the sub-files according to the sampling ratio to obtain the sampled data and determines the target sharding ratio of the sub-files according to the sharding results of the sampled data, thereby avoiding traversing the entire sub-file, improving the efficiency of determining the target sharding ratio of the sub-files, and reducing the performance loss caused by the data sampling process. Dynamically adjusting the target migration capacity of different data nodes according to the data migration speed of different data nodes and the resource usage of the physical machines of the data nodes avoids transmission delays and blockages caused by excessive data volume during the migration process, thereby shortening the migration time. According to the target migration capacity, sub-files are selected from the candidate file ratio queue as sub-files that can be migrated concurrently, and the migration process of the distributed database is optimized from the data distribution level of the data files to be migrated, thereby improving the efficiency of migration on the basis of fully utilizing the data node resources.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a first flow chart of a distributed data table migration method provided by an embodiment of the present invention;

[0025] Figure 2 is a second flow chart of the distributed data table migration method provided by an embodiment of the present invention;

[0026] Figure 3 It is a structural schematic diagram of a distributed data table migration device provided by an embodiment of the present invention;

[0027] Figure 4 It is a schematic diagram of the structure of an electronic device implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Figure 1 This is the first flow chart of the distributed data table migration method provided by the embodiment of the present invention. This embodiment is applicable to the case of migrating data to a distributed database. The method can be executed by a distributed data table migration device, which can be implemented in the form of hardware and / or software. The device can be configured in an electronic device with corresponding data processing capabilities. Figure 1 As shown, the method includes:

[0031] S110: Cut the source table data into multiple sub-files.

[0032] During the data migration process, if the source data table is a very large table, the exported source table data will not be stored in one file, but will be cut into multiple sub-files according to certain rules to avoid the import of very large files. For example, the source table can be cut into multiple sub-files according to the number of data rows or the size of the file. When migrating the source table data to the target data table, you only need to migrate the multiple sub-files corresponding to the source table to the target data table.

[0033] Optionally, the cutting of the source table data to obtain a plurality of sub-files includes: cutting the source table data to obtain a plurality of sub-files, the number of source tables is greater than or equal to 1, and each source table corresponds to a sampling queue for storing the sub-files of the current source table.

[0034] Multiple source table data can be migrated to the target data table. Each source table corresponds to a sampling queue, which stores multiple sub-files obtained by cutting the current source table. By setting a corresponding sampling queue for each source table, it is ensured that the sub-files in the same sampling queue come from the same source table data, thus having similar data distribution.

[0035] S120: Sampling the sub-files according to the sampling ratio to obtain sampling data.

[0036] S130: Slice the sampled data using the slicing rule and slicing key of the target data table to obtain a target slicing ratio of sub-files.

[0037] The target data table is stored on at least one data node, that is, the target data table is stored in a distributed database, the sharding rule and sharding key of the target data table are obtained, and the sampled data is sharded according to the sharding rule and sharding key to obtain the sharding ratio of the sampled data as the target sharding ratio of the current sub-file. The shards correspond to the data nodes, indicating that the data in the current sub-file will be stored on the corresponding data nodes according to the target sharding ratio. The sub-files of each source table usually have the characteristics of a large number of files and a large data capacity of the files. By sampling the sub-files according to the sampling ratio to obtain the sampled data, the target sharding ratio of the sub-file is determined according to the sharding result of the sampled data, which avoids traversing the entire sub-file and improves the efficiency of determining the target sharding ratio of the sub-file.

[0038] In an optional implementation, the method further includes: adjusting the sampling ratio according to a target sharding ratio of a previous subfile and a target sharding ratio of a current subfile.

[0039] The subfiles in the same sampling queue belong to the same source table, and the subfiles of the same source table have similar data distribution. Therefore, the sampling ratio can be adjusted according to the target sharding ratio of the previous subfile and the target sharding ratio of the current subfile. The subfiles are sampled according to the adjusted sampling ratio to obtain sampling data, and the target sharding ratio of the subfile is determined according to the sharding result of the sampling data, which can improve the accuracy and efficiency of determining the target sharding ratio of the subfile.

[0040] Optionally, the sampling ratio is adjusted according to the target sharding ratio of the previous subfile and the target sharding ratio of the current subfile, including: if the target sharding ratio of the previous subfile is the same as the target sharding ratio of the current subfile, then the sampling ratio is reduced; if the target sharding ratio of the previous subfile is different from the target sharding ratio of the current subfile, then the sampling ratio is increased. When the target sharding ratio of the previous subfile is the same as the target sharding ratio of the current subfile, it indicates that the data distribution in different subfiles is similar and the source table data is evenly distributed. At this time, the sampling ratio is reduced, and the sharding ratio of the subfile can be accurately reflected with less sampled data. When the target sharding ratio of the previous subfile is different from the target sharding ratio of the current subfile, it indicates that there are differences in the data distribution in different subfiles and the source table data is not evenly distributed. At this time, the sampling ratio needs to be increased, and the sampled data needs to be increased so that the sharding result of the sampled data can accurately reflect the sharding ratio of the subfile.

[0041] For example, the subfile is sampled according to the default sampling ratio of 0.3 to obtain sampled data. The target data table is stored in four data nodes of the distributed database. The sampled data is sharded according to the sharding rule and sharding key of the target data table. The sharding ratio of the sampled data corresponding to the four data nodes is 1:1:0:1, and the target sharding ratio of the subfile is determined to be 1:1:0:1. The next subfile in the current sampling queue is sampled according to the default sampling ratio of 0.3 to obtain sampled data. The sampled data is sharded according to the sharding rule and sharding key of the target data table. The sharding ratio of the sampled data corresponding to the four data nodes is 1:1:0:1, and the target sharding ratio of the current subfile is determined to be 1:1:0:1. The target sharding ratio of the current subfile is the same as the target sharding ratio of the previous subfile. The sampling ratio is adjusted to 0.1. When sampling the next subfile in the current sampling queue, sampling is performed according to the adjusted sampling ratio of 0.1. The next subfile in the current sampling queue is sampled according to the adjusted sampling ratio of 0.1 to obtain sampled data. The sampled data is sharded according to the sharding rule and sharding key of the target data table, and the sharding ratio of the sampled data corresponding to the four data nodes is 1:0:0:1. The target sharding ratio of the current subfile is determined to be 1:0:0:1. The target sharding ratio of the current subfile is different from the target sharding ratio of the previous subfile, so the sampling ratio is adjusted to 0.2. By adjusting the sampling ratio according to the target sharding ratio of the previous subfile and the target sharding ratio of the current subfile, the performance loss caused by the data sampling process is reduced while the accuracy of determining the target sharding ratio of the subfile is improved.

[0042] S140: Based on the shard ratio associated with the candidate file ratio queue, determine the target file ratio queue of the subfile from the candidate file ratio queue according to the target shard ratio, and store the subfile in the target file ratio queue.

[0043] A candidate file ratio queue is set according to the number of data nodes and an associated shard ratio is set for the candidate file ratio queue, and each candidate file ratio queue is associated with a shard ratio. If the target shard ratio of a subfile is the same as the shard ratio associated with any candidate file ratio queue, the candidate file ratio queue is determined as the target file ratio queue for the subfile, and the subfile is stored in the target file ratio queue.

[0044] Exemplarily, when the target data table is stored in four data nodes of a distributed database, 15 candidate file ratio queues and their associated shard ratios may be configured as follows: (candidate file ratio queue one, 1:0:0:0), (candidate file ratio queue two, 0:1:0:0), (candidate file ratio queue three, 0:0:1:0), (candidate file ratio queue four, 0:0:0:1), (candidate file ratio queue five, 1:1:0:0), (candidate file ratio queue six, 1:0:1:0), (candidate file ratio queue five, 1:1:0:0), (candidate file ratio queue six, 1:0:1:0), (candidate file ratio queue seven, 1:0:1:0), (candidate file ratio queue eight ... Select file proportion queue seven, 1:0:0:1), (candidate file proportion queue eight, 0:1:1:0), (candidate file proportion queue nine, 0:1:0:1), (candidate file proportion queue ten, 0:0:1:1), (candidate file proportion queue eleven, 1:1:1:0), (candidate file proportion queue twelve, 1:1:0:1), (candidate file proportion queue thirteen, 1:0:1:1), (candidate file proportion queue fourteen, 0:1:1:1), (candidate file proportion queue fifteen, 1:1:1:1). When the target sharding ratio of the sub-file is 1:0:0:1, the sub-file is stored in candidate file proportion queue seven.

[0045] S150: According to the candidate file proportion queue, the data capacity of the sub-file, the first migration queue, and the second migration queue, select a sub-file from the candidate file proportion queue and move the sub-file to the first migration queue.

[0046] Optionally, the second migration queue is the first migration queue when the performance of the data node reaches a performance threshold.

[0047] S160: Store the sub-files in the first migration queue in parallel to a target data table.

[0048] In the initial migration phase, subfiles are selected from the candidate file proportion queue, and the target shard ratio of each selected subfile is summed up according to the corresponding shards to obtain the migration shard ratio, so that the migration shard ratio corresponding to each data node is the same. For example, 5 subfiles are selected from (candidate file proportion queue 1, 1:0:0:0), 5 subfiles are selected from (candidate file proportion queue 2, 0:1:0:0), 5 subfiles are selected from (candidate file proportion queue 3, 0:0:1:0), and 5 subfiles are selected from (candidate file proportion queue 4, 0:0:0:1). The migration shard ratio obtained according to each selected subfile is 1:1:1:1. The selected 20 subfiles are migrated from the candidate file proportion queue to the first migration queue, so that the pressure on each data node is the same when the subfiles in the first migration queue are stored in parallel to the target data table.

[0049] Monitor the performance status of the data nodes in real time. When the performance of the data nodes reaches the performance threshold, for example, the CPU utilization of the data nodes reaches 90% and / or the memory usage reaches 90%, use the first migration queue as the second migration queue, and determine the shard data capacity of each data node corresponding to each sub-file in the second migration queue according to the shard ratio associated with the candidate file proportion queue and the data capacity of the sub-files. Determine the data capacity of each shard as the maximum migration capacity of the data node based on the shard data capacity of each sub-file corresponding to each data node.

[0050] During the migration process, at certain time intervals, the shard data capacity of each data node corresponding to each sub-file in the first migration queue is determined according to the shard ratio associated with the candidate file ratio queue and the data capacity of the sub-file, and the data capacity of each shard is determined as the used migration capacity of the data node according to the shard data capacity of each sub-file corresponding to each data node. According to the maximum migration capacity and used migration capacity of the data node, the available migration capacity of the data node is determined as the target migration capacity, and sub-files are selected from the candidate file ratio queue according to the target migration capacity of each data node, so that the data capacity of each shard corresponding to the selected sub-file does not exceed the target migration capacity of the corresponding data node. The sub-files are moved to the first migration queue, and the sub-files in the first migration queue are stored in parallel to the target data table.

[0051] The maximum migration capacity of the data node is determined according to the candidate file ratio queue, the data capacity of the sub-file and the second migration queue; the used migration capacity of the data node is determined according to the candidate file ratio queue, the data capacity of the sub-file and the first migration queue, so as to determine the target migration capacity of each data node, and according to the target migration capacity of the data node, the sub-file is selected from the candidate file ratio queue so that the data capacity of each shard corresponding to the selected sub-file does not exceed the target migration capacity of the corresponding data node. The target migration capacity of different data nodes can be dynamically adjusted according to the data migration speed of different data nodes and the resource usage of the physical machine of the data node, avoiding transmission delays and blockages caused by excessive data volume during the migration process, thereby shortening the migration time. Selecting sub-files from the candidate file ratio queue as files that can be migrated concurrently according to the target migration capacity can make full use of the bandwidth and storage resources of the data node, avoid idle or overused resources, improve resource utilization efficiency, and thus improve data migration efficiency.

[0052] The embodiment of the present invention samples the subfiles according to the sampling ratio to obtain the sampled data, and determines the target sharding ratio of the subfile according to the sharding result of the sampled data, thereby avoiding traversing the entire subfile, improving the efficiency of determining the target sharding ratio of the subfile, and reducing the performance loss caused by the data sampling process. Further, by adjusting the sampling ratio according to the target sharding ratio of the previous subfile and the target sharding ratio of the current subfile, the accuracy of determining the target sharding ratio of the subfile is improved while reducing the performance loss caused by the data sampling process. Dynamically adjusting the target migration capacity of different data nodes according to the data migration speed of different data nodes and the resource usage of the physical machine of the data node avoids transmission delays and blockages caused by excessive data volume during the migration process, thereby shortening the migration time. Selecting subfiles from the candidate file ratio queue as files that can be migrated concurrently according to the target migration capacity can make full use of the bandwidth and storage resources of the data node, avoid idle or overused resources, improve resource utilization efficiency, and thus improve data migration efficiency.

[0053] Figure 2 2 is a second flow chart of the distributed data table migration method provided by an embodiment of the present invention. This embodiment is optimized and improved on the basis of the above embodiment. Figure 2 As shown, the method includes:

[0054] S210: Cut the source table data into multiple sub-files.

[0055] S220: Sampling the sub-files according to the sampling ratio to obtain sampling data.

[0056] S230: Slice the sampled data using the slicing rule and slicing key of the target data table to obtain a target slicing ratio of the sub-files.

[0057] The target data table is stored on at least one data node.

[0058] S240: Based on the shard ratio associated with the candidate file ratio queue, determine the target file ratio queue of the subfile from the candidate file ratio queue according to the target shard ratio, and store the subfile in the target file ratio queue.

[0059] S250. Determine a first multiple according to the shard ratio associated with the candidate file proportion queue, and convert the target shard ratio of the sub-file into an alignment ratio according to the first multiple; divide the alignment ratio by the first multiple and multiply by the data capacity of the sub-file to obtain the shard data capacity of the sub-file.

[0060] Calculate the least common multiple of the shard ratios associated with the candidate file ratio queue as the first multiple. Convert the shard ratios associated with the candidate file ratio queue into alignment ratios according to the first multiple. Specifically, multiply each shard ratio in the shard ratio by the first multiple to obtain the alignment ratio. Convert the target shard ratio of the sub-file into alignment ratios according to the first multiple, that is, multiply each target shard ratio in the target shard ratio by the first multiple to obtain the alignment ratio. Divide the alignment ratio by the first multiple and multiply by the data capacity of the sub-file to obtain the shard data capacity of the sub-file.

[0061] Continuing with the above example, the target data table is stored in four data nodes of the distributed database. (Candidate file proportion queue 1, 1:0:0:0), (Candidate file proportion queue 2, 0:1:0:0), (Candidate file proportion queue 3, 0:0:1:0), (Candidate file proportion queue 4, 0:0:0:1), (Candidate file proportion queue 5, 1:1:0:0), (Candidate file proportion queue 6, 1:0:1:0), (Candidate file proportion queue 7, 1:0:0:1), (Candidate file proportion queue 8, 0:1:1:0), (Candidate file proportion queue 9, 0:1:1:0), (Candidate file proportion queue 10, 0:1:1:0), (Candidate file proportion queue 11, 0:1:1:0), (Candidate file proportion queue 12, 0:1:1:0), (Candidate file proportion queue 13, 0:1:1:0), (Candidate file proportion queue 14, 0:0:0:1), (Candidate file proportion queue 15, 1:1:0:0), (Candidate file proportion queue 16, 1:0:1:0), (Candidate file proportion queue 17, 1:0:0:1), (Candidate file proportion queue 18, 0:1:1:0), (Candidate file proportion queue 19, 0:1:1:0), (Candidate file proportion queue 20, 0:1:1:0), (Candidate file proportion queue 21, 0:1:1:0), (Candidate file proportion queue 22, 0:1:1:0), (Candidate file proportion queue 23, 0:1:1:0), (Candidate file proportion queue 24, 0:1:1:0), (Candidate file proportion queue 25, 0:1:1 The least common multiple of (1, 2, 3, 4) is 12. The shard ratio associated with the candidate file ratio queue is converted into an alignment ratio according to the first multiple, that is, each shard ratio in the shard ratio is multiplied by the first multiple to obtain the alignment ratio. For example, when the shard ratio is 1:1:0:0, the shard ratio of shard one is 1 / 2, and the value of shard one in the converted alignment ratio is (1 / 2)*12=6. After conversion, the candidate file proportion queues are (candidate file proportion queue one, 12:0:0:0), (candidate file proportion queue two, 0:12:0:0), (candidate file proportion queue three, 0:0:12:0), (candidate file proportion queue four, 0:0:0:12), (candidate file proportion queue five, 6:6:0:0), (candidate file proportion queue six, 6:0:6:0), (candidate file proportion queue seven, 6:0:0:6), (candidate file proportion queue eight, 0:6:6:0), (candidate file proportion queue nine, 0:6:0:6), (candidate file proportion queue ten, 0:0:6:6), (candidate file proportion queue eleven, 4:4:4:0), (candidate file proportion queue twelve, 4:4:0:4), (candidate file proportion queue thirteen, 4:0:4:4), (candidate file proportion queue fourteen, 0:4:4:4), (candidate file proportion queue fifteen, 3:3:3:3). The target sharding ratio of the subfile is 1:1:0:1, and the alignment ratio of the subfile after conversion is 4:4:0:4. If the data capacity of the subfile is 60W rows, the sharding data capacity of the subfile corresponding to each data node is: the sharding data capacity of data node one is (4 / 12)*60W=20W rows. Similarly, the sharding data capacity of data node two is 20W rows, the sharding data capacity of data node three is 0, and the sharding data capacity of data node four is 20W rows.

[0062] S260: Determine the used migration capacity of each data node according to the shard data capacity of the sub-files in the first migration queue; and determine the maximum migration capacity of each data node according to the shard data capacity of the sub-files in the second migration queue.

[0063] The second migration queue is the first migration queue when the performance of the data node reaches a performance threshold.

[0064] S270: Determine a target migration capacity of the data node according to the maximum migration capacity and the used migration capacity of the data node, select a sub-file from the candidate file proportion queue according to the target migration capacity, and move the sub-file to the first migration queue.

[0065] Determine the shard data capacity of each data node corresponding to each sub-file in the first migration queue, sum the shard data capacity of each data node corresponding to each sub-file by shard, and determine the used migration capacity of each data node. Exemplarily, if the sub-files in the first migration queue are (sub-file one 1:1:0:1), (sub-file two 1:0:0:1), and (sub-file three 1:1:0:0), the sub-file data capacity is 60W rows, the alignment ratio of sub-file one is 4:4:0:4, and the shard data capacity of sub-file one corresponding to data node one is (4 / 12)*60W=20W rows. The shard data capacity of each data node corresponding to each sub-file in the first migration queue is (sub-file one 20W, 20W, 0, 20W), (sub-file two 30W, 0, 0, 30W), and (sub-file three 30W, 30W, 0, 0); the shard data capacity of each sub-file corresponding to each data node is summed according to the shards, and it is determined that the used migration capacity of the data nodes is (80W, 50W, 0, 50W) respectively.

[0066] When the performance of the data node reaches the performance threshold, the first migration queue is used as the second migration queue, and the maximum migration capacity of each data node is determined according to the shard data capacity of the sub-files in the second migration queue. Specifically, the shard data capacity of each data node corresponding to each sub-file in the second migration queue is determined, and the shard data capacity of each data node corresponding to each sub-file is summed according to the shards to determine the maximum migration capacity of each data node.

[0067] According to the maximum migration capacity and used migration capacity of the data node, the available migration capacity of the data node is determined as the target migration capacity, and sub-files are selected from the candidate file ratio queue according to the target migration capacity of each data node, so that the data capacity of each shard corresponding to the selected sub-file does not exceed the target migration capacity of the corresponding data node. Continuing with the above example, the used migration capacity of the data node is (80W, 50W, 0, 50W) respectively. If the maximum migration capacity of the data node is (100W, 100W, 80W, 80W) respectively, then the target migration capacity of the data node is determined to be (20W, 50W, 80W, 30W). Select sub-files from the candidate file ratio queue so that the data capacity of each shard corresponding to the selected sub-file does not exceed the target migration capacity of the corresponding data node (20W, 50W, 80W, 30W). Move each selected sub-file from the candidate file ratio queue to the first migration queue.

[0068] In an optional implementation, selecting sub-files from the candidate file proportion queue according to the target migration capacity and moving the sub-files to the first migration queue includes: selecting sub-files from the candidate file proportion queue according to the maximum value in the target migration capacity; when the sum of the shard data capacity corresponding to each data node in each selected sub-file does not exceed the target migration capacity of the corresponding data node, moving each selected sub-file from the candidate file proportion queue to the first migration queue, and updating the target migration capacity.

[0069] According to the maximum value in the target migration capacity, select sub-files from the candidate file ratio queue. Specifically, use the data node corresponding to the maximum value in the target migration capacity as the reference data node, so that the shard data capacity corresponding to the reference data node in each selected sub-file is less than or equal to the target migration capacity of the reference data node, and at the same time, the sum of the shard data capacities corresponding to other data nodes in each selected sub-file does not exceed the target migration capacity of the corresponding data node. Move each selected sub-file from the candidate file ratio queue to the first migration queue to update the first migration queue, update the target migration capacity according to the updated first migration queue, and continue to select sub-files from the candidate file ratio queue according to the target migration capacity, until the sum of the shard data capacity corresponding to each data node in each selected sub-file cannot exceed the target migration capacity of the corresponding data node, and end the sub-file selection process.

[0070] Exemplarily, the fragment data capacity of the subfiles in each candidate file proportion queue is respectively (candidate file proportion queue one, 12:0:0:0, (60W, 0, 0, 0)), (candidate file proportion queue two, 0:12:0:0, (0, 60W, 0, 0)), (candidate file proportion queue three, 0:0:12:0, (0, 0, 60W, 0)), (candidate file proportion queue four, 0:0:0:12, (0, 0, 0, 60W)), (candidate file proportion queue five, 6:6:0:0, (30W, 30W, 0, 0)), (candidate file proportion queue six, 6:0:6:0, (30W, 0, 30W, 0)), (Candidate file share queue seven, 6:0:0:6, (30W, 0, 0, 30W)), (Candidate file share queue eight, 0:6:6:0, (0, 30W, 30W, 0)), (Candidate file share queue nine, 0:6:0:6, (0, 30W, 0, 30W)), (Candidate file share queue ten, 0:0:6:6, (0, 0, 30W, 30W)), (Candidate file share queue eleven, 4:4:4:0, (20W, 20W, 20W, 0)), (Candidate file share queue twelve ,4:4:0:4,(20W,20W,0,20W)),(candidate file proportion queue thirteen,4:0:4:4,(20W,0,20W,20W),(candidate file proportion queue fourteen,0:4:4:4,(0,20W,20W,20W),(candidate file proportion queue fifteen,3:3:3:3,(15W,15W,15W,15W)). The maximum value among the target migration capacity (20W,50W,80W,30W) is 80W. According to the maximum value among the target migration capacity, a subfile can be selected from the candidate file proportion queue. Select a subfile from the proportion queue three and select a subfile from the candidate file proportion queue eleven. At this time, the sum of the shard data capacity corresponding to each data node in the selected subfiles is (20W, 20W, 80W, 0). At this time, the sum of the shard data capacity corresponding to each data node in the selected subfiles does not exceed the target migration capacity of the corresponding data node. Therefore, select a subfile from the candidate file proportion queue three and select a subfile from the candidate file proportion queue eleven. Move the selected two subfiles from the candidate file proportion queue to the first migration queue to update the first migration queue.According to the updated first migration queue, the target migration capacity is updated to (0, 30W, 0, 30W), and sub-files are continued to be selected from the candidate file proportion queue according to the target migration capacity. According to the target migration capacity, a sub-file can be selected from candidate file proportion queue nine and moved from candidate file proportion queue nine to the first migration queue, and the target migration capacity is updated to (0, 0, 0, 0). At this time, it cannot be satisfied that the sum of the shard data capacity corresponding to each data node in the selected sub-files does not exceed the target migration capacity of the corresponding data node, and the sub-file selection process is ended.

[0071] By determining a first multiple according to the shard ratio associated with the candidate file proportion queue, and converting the target shard ratio of the subfile into an alignment ratio according to the first multiple; dividing the alignment ratio by the first multiple and multiplying the data capacity of the subfile to obtain the shard data capacity of the subfile. The shard data capacity corresponding to each data node in the subfile can be accurately estimated. The used migration capacity of each data node is determined according to the shard data capacity of the subfile in the first migration queue; the maximum migration capacity of each data node is determined according to the shard data capacity of the subfile in the second migration queue. The target migration capacity of the data node is determined according to the maximum migration capacity and the used migration capacity of the data node, and the subfile is selected from the candidate file proportion queue according to the target migration capacity and moved to the first migration queue. The migration pressure caused by the subfile on each data node can be accurately judged, so that when data is migrated, the concurrency of the migration file can be dynamically adjusted according to the distribution of the data to be migrated, avoiding the problem that the migration pressure is concentrated on a certain data node when the data is unevenly distributed in the data file to be migrated, resulting in unreasonable resource utilization and low migration efficiency. The migration process of the distributed database is optimized from the data distribution level of the data files to be migrated, and the efficiency of migration is improved on the basis of fully utilizing the data node resources.

[0072] S280: Store the sub-files in the first migration queue in parallel to a target data table.

[0073] The sub-files in the first migration queue are stored in parallel to the target data table through concurrent threads. The used migration capacity of each data node is determined at a first time interval based on the shard data capacity of the sub-files in the first migration queue. The maximum migration capacity of each data node is determined at a second time interval based on the shard data capacity of the sub-files in the second migration queue. The second time interval is greater than the first time interval. The file migration speed on different data nodes, the shard data capacity of the sub-files in the first migration queue, and the resource occupancy of the physical machine where the data node is located are monitored to determine the performance status of the data node, and the maximum migration capacity is adjusted in time according to the performance status of the data node to optimize the migration concurrency of the first migration queue.

[0074] By regularly evaluating and adjusting the maximum migration capacity of data nodes, existing resources can be used more effectively, avoiding idle or overused resources, thereby improving resource utilization. When the physical node where the data node is located fails, timely adjusting the maximum migration capacity can avoid the inefficiency of the entire migration task caused by transmission delays and blockages caused by the physical node failure. By regularly evaluating and adjusting the maximum migration capacity, the data migration process can flexibly adapt to the changes of data nodes, and can also help system administrators allocate and optimize resources more reasonably. For example, for data nodes with heavy loads, you can consider increasing their storage capacity or improving their processing power; for data nodes with light loads, you can use their resources for other tasks or as backup nodes. This optimization of resource allocation can not only improve the overall performance of the system, but also reduce operating costs.

[0075] The embodiment of the present invention determines a first multiple according to the shard ratio associated with the candidate file ratio queue, and converts the target shard ratio of the subfile into an alignment ratio according to the first multiple; the alignment ratio is divided by the first multiple and multiplied by the data capacity of the subfile to obtain the shard data capacity of the subfile. The shard data capacity corresponding to each data node in the subfile can be accurately estimated. The used migration capacity of each data node is determined according to the shard data capacity of the subfile in the first migration queue; the maximum migration capacity of each data node is determined according to the shard data capacity of the subfile in the second migration queue. The target migration capacity of the data node is determined according to the maximum migration capacity and the used migration capacity of the data node, and the subfile is selected from the candidate file ratio queue according to the target migration capacity and moved to the first migration queue. The migration pressure caused by the subfile on each data node can be accurately judged, so that when data is migrated, the concurrency of the migration file can be dynamically adjusted according to the distribution of the data to be migrated, avoiding the problem that the migration pressure that may occur when the data is unevenly distributed in the data file to be migrated is concentrated on a certain data node, resulting in unreasonable resource utilization and low migration efficiency. The migration process of the distributed database is optimized from the data distribution level of the data files to be migrated, and the migration efficiency is improved on the basis of making full use of the data node resources; transmission delays and blockages caused by excessive data volume during the migration process are avoided, thereby shortening the migration time. Selecting sub-files from the candidate file ratio queue as files that can be migrated concurrently according to the target migration capacity can make full use of the bandwidth and storage resources of the data node, avoid idle or overused resources, improve resource utilization efficiency, and thus improve data migration efficiency.

[0076] Figure 3 It is a structural schematic diagram of a distributed data table migration device provided by an embodiment of the present invention.

[0077] like Figure 3 As shown, the device comprises:

[0078] The source data segmentation module 310 is used to segment the source table data to obtain multiple sub-files;

[0079] The sub-file sharding ratio module 320 is used to sample the sub-files according to the sampling ratio to obtain sampled data; slice the sampled data using the sharding rule and sharding key of the target data table to obtain a target sharding ratio of the sub-files; the target data table is stored on at least one data node;

[0080] The sub-file division module 330 is used to determine the target file share queue of the sub-file from the candidate file share queue according to the target share ratio based on the slice ratio associated with the candidate file share queue, and store the sub-file in the target file share queue;

[0081] The sub-file migration module 340 is used to select a sub-file from the candidate file ratio queue and move the sub-file to the first migration queue according to the candidate file ratio queue, the data capacity of the sub-file and the first migration queue, and the second migration queue; and store the sub-files in the first migration queue in parallel to the target data table.

[0082] The distributed data table migration device provided in the embodiment of the present invention can execute the distributed data table migration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0083] Optionally, the device further comprises:

[0084] The sampling ratio adjustment module is used to adjust the sampling ratio according to the target fragmentation ratio of the previous sub-file and the target fragmentation ratio of the current sub-file.

[0085] Optionally, the sampling ratio adjustment module is specifically used to reduce the sampling ratio if the target sharding ratio of the previous subfile is the same as the target sharding ratio of the current subfile; if the target sharding ratio of the previous subfile is different from the target sharding ratio of the current subfile, then increase the sampling ratio.

[0086] Optional, sub-file migration module, including:

[0087] A migration capacity determination unit, configured to determine a first multiple according to the shard ratio associated with the candidate file ratio queue, and convert the target shard ratio of the sub-file into an alignment ratio according to the first multiple; obtain the shard data capacity of the sub-file by dividing the alignment ratio by the first multiple and multiplying by the data capacity of the sub-file; determine the used migration capacity of each data node according to the shard data capacity of the sub-file in the first migration queue; determine the maximum migration capacity of each data node according to the shard data capacity of the sub-file in the second migration queue;

[0088] The sub-file selection unit is used to determine the target migration capacity of the data node according to the maximum migration capacity and the used migration capacity of the data node, and select a sub-file from the candidate file proportion queue according to the target migration capacity and move the sub-file to the first migration queue.

[0089] The second migration queue is the first migration queue when the performance of the data node reaches a performance threshold.

[0090] Optionally, the sub-file selection unit is specifically used to select a sub-file from the candidate file proportion queue according to the maximum value in the target migration capacity; when the sum of the shard data capacity corresponding to each data node in the selected sub-files does not exceed the target migration capacity of the corresponding data node, the selected sub-files are moved from the candidate file proportion queue to the first migration queue, and the target migration capacity is updated.

[0091] Optionally, the source data segmentation module is specifically used to segment source table data to obtain multiple sub-files, the number of source tables is greater than or equal to 1, and each source table corresponds to a sampling queue for storing sub-files of the current source table.

[0092] The distributed data table migration device further described can also execute the distributed data table migration method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0093] Figure 4 A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0094] like Figure 4As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0095] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0096] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 executes the various methods and processes described above, such as a distributed data table migration method.

[0097] In some embodiments, the distributed data table migration method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the distributed data table migration method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the distributed data table migration method in any other appropriate manner (e.g., by means of firmware).

[0098] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0099] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0100] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0101] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0102] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0103] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0104] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0105] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed data table migration method, characterized in that: The method comprises: Cut the source table data into multiple sub-files; Sampling the sub-files according to the sampling ratio to obtain sampling data; The sampled data is sliced ​​using a slicing rule and a slicing key of a target data table to obtain a target slicing ratio of subfiles; the target data table is stored on at least one data node; Based on the shard ratio associated with the candidate file ratio queue, determine the target file ratio queue of the subfile from the candidate file ratio queue according to the target shard ratio, and store the subfile in the target file ratio queue; According to the candidate file proportion queue, the data capacity of the sub-file, the first migration queue, and the second migration queue, selecting a sub-file from the candidate file proportion queue and moving the sub-file to the first migration queue; The sub-files in the first migration queue are stored in parallel in the target data table.

2. The method according to claim 1, characterized in that The method further comprises: The sampling ratio is adjusted according to the target sharding ratio of the previous subfile and the target sharding ratio of the current subfile.

3. The method according to claim 2, characterized in that The adjusting the sampling ratio according to the target sharding ratio of the previous subfile and the target sharding ratio of the current subfile includes: If the target sharding ratio of the previous subfile is the same as the target sharding ratio of the current subfile, reducing the sampling ratio; If the target sharding ratio of the previous subfile is different from the target sharding ratio of the current subfile, the sampling ratio is increased.

4. The method according to claim 1, characterized in that: The selecting a sub-file from the candidate file proportion queue and moving the sub-file to the first migration queue according to the candidate file proportion queue, the data capacity of the sub-file, the first migration queue, and the second migration queue includes: Determine a first multiple according to the shard ratio associated with the candidate file proportion queue, and convert the target shard ratio of the sub-file into an alignment ratio according to the first multiple; The segment data capacity of the sub-file is obtained by dividing the alignment ratio by the first multiple and multiplying by the data capacity of the sub-file; Determine the used migration capacity of each data node according to the shard data capacity of the sub-files in the first migration queue; Determine the maximum migration capacity of each data node according to the shard data capacity of the sub-files in the second migration queue; The target migration capacity of the data node is determined according to the maximum migration capacity and the used migration capacity of the data node, and a sub-file is selected from the candidate file proportion queue according to the target migration capacity and the sub-file is moved to the first migration queue.

5. The method according to claim 1, characterized in that The second migration queue is the first migration queue when the performance of the data node reaches a performance threshold.

6. The method according to claim 4, characterized in that The selecting a sub-file from the candidate file proportion queue according to the target migration capacity and moving the sub-file to the first migration queue includes: Selecting a subfile from the candidate file proportion queue according to the maximum value in the target migration capacity; When the sum of the shard data capacity corresponding to each data node in the selected sub-files does not exceed the target migration capacity of the corresponding data node, the selected sub-files are moved from the candidate file proportion queue to the first migration queue, and the target migration capacity is updated.

7. The method according to claim 1, characterized in that The source table data is cut to obtain multiple sub-files, including: The source table data is cut to obtain a plurality of sub-files, the number of the source tables is greater than or equal to 1, and each source table corresponds to a sampling queue for storing the sub-files of the current source table.

8. A distributed data table migration device, characterized in that: The device comprises: The source data segmentation module is used to segment the source table data into multiple sub-files; A sub-file sharding ratio module is used to sample sub-files according to a sampling ratio to obtain sampled data; slice the sampled data using a sharding rule and a sharding key of a target data table to obtain a target sharding ratio of the sub-files; the target data table is stored on at least one data node; A sub-file division module, configured to determine a target file proportion queue of the sub-file from the candidate file proportion queue according to the target shard proportion based on the shard proportion associated with the candidate file proportion queue, and store the sub-file in the target file proportion queue; The sub-file migration module is used to select a sub-file from the candidate file ratio queue and move the sub-file to the first migration queue according to the candidate file ratio queue, the data capacity of the sub-file and the first migration queue, and the second migration queue; and store the sub-files in the first migration queue in parallel to the target data table.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the distributed data table migration method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the distributed data table migration method according to any one of claims 1 to 7 when executed.

Images