Data migration method, device and storage medium

Abstract

The present application discloses a data migration method, a device and a storage medium, the data migration method comprising: dividing a data table into a plurality of partition tables based on a value range of preset column data in a data table stored in a source cluster; sorting each partition table according to the migration priority of each partition table to obtain a sorting result of each partition table; obtaining the migration data volume corresponding to the current batch, and selecting the partition tables to be migrated in the current batch from the sorting result; wherein the total data volume of the selected partition tables is less than or equal to the migration data volume; based on the migration priorities respectively corresponding to the partition tables to be migrated in the current batch, migrating the data in the partition tables to be migrated in the current batch to the target cluster in sequence, so as to give priority to migrating data with high priority, thereby ensuring the effective operation of some businesses, reducing the impact of data migration on businesses, and avoiding migration failures caused by insufficient memory by migrating data in batches.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data migration method, device and storage medium. Background Art

[0002] With the development of computer technology, databases have been widely used in all walks of life. A database is a warehouse that organizes, stores, and manages data according to data structures. It is a collection of large amounts of data that is stored in a computer for a long time, organized, shareable, and uniformly managed.

[0003] In actual use, it is necessary to migrate all or part of the data in the database cluster to another cluster. The current migration method still has problems with low success rate and migration efficiency. For example, if the amount of data is large during the full migration, the migration time will be very long, and the new cluster will be unavailable during the migration. If it is a real-time business, it will affect the business use for a long time. In addition, when performing data migration, all data in the table needs to be loaded into the memory, which is prone to failure of the migration task due to insufficient memory. If the migration fails multiple times, it will seriously affect the data use. Summary of the invention

[0004] In order to solve the above technical problems, the present application at least provides a data migration method, device and storage medium.

[0005] In a first aspect, the present application provides a data migration method, which is applied to a source cluster, and the source cluster stores data through a data table. The method includes: dividing the data table into multiple partition tables based on the value range of preset column data in the data table stored in the source cluster; comparing the order relationship between the values ​​of the preset column data in each partition table to obtain the migration priority of each partition table, and sorting each partition table according to the migration priority of each partition table to obtain the sorting result of each partition table; obtaining the migration data volume corresponding to the current batch, and selecting the partition tables to be migrated in the current batch from the sorting results; wherein the total data volume of the selected partition tables is less than or equal to the migration data volume; based on the migration priorities corresponding to the partition tables to be migrated in the current batch, the data in the partition tables to be migrated in the current batch are migrated to the target cluster in sequence.

[0006] In one embodiment, based on the value range of preset column data in the data table stored in the source cluster, the data table is divided into multiple partition tables, including: obtaining the range granularity for dividing the data table, the range granularity is used to limit the range size of the value range; determining the partition start value and the partition end value of the value range based on the range granularity; and dividing the data rows in the data table where the preset column data is within the partition start value and the partition end value into the partition table corresponding to the value range.

[0007] In one embodiment, obtaining the range granularity for dividing the data table includes: obtaining data read and write operations for data table records; and setting the range granularity for dividing the data table based on the data read and write operations.

[0008] In one embodiment, the range granularity for dividing the data table is set based on data read and write operations, including: counting the number of data read and write operations within a preset time period to obtain statistical results; and calculating the range granularity for dividing the data table based on the statistical results.

[0009] In one embodiment, the range granularity of dividing the data table is set based on the data read and write operations, including: obtaining the fields and services associated with each data read and write operation, and statistically obtaining the services associated with each field; based on the importance corresponding to the services associated with each field, obtaining the range granularity that matches the data table.

[0010] In one embodiment, the preset column data includes time data, and the value range includes a time range; based on the value range of the preset column data in the data table stored in the source cluster, the data table is divided into multiple partition tables, including: based on the time range of the time data in the data table stored in the source cluster, the data table is divided into multiple partition tables.

[0011] In one embodiment, obtaining the migration data volume corresponding to the current batch includes: detecting available memory resources of the source cluster and / or the target cluster; and setting a single migration data volume based on the available memory resources.

[0012] In one embodiment, detecting the available memory resources of the source cluster and / or the target cluster includes: recording the memory resource usage of the source cluster and / or the target cluster in a historical time period; analyzing the memory resource change trend of the source cluster and / or the target cluster in a future time period based on the memory resource usage in the historical time period; and calculating the available memory resources of the source cluster and / or the target cluster based on the memory resource change trend.

[0013] The second aspect of the present application provides a data migration device, which includes: a table partitioning module, which is used to divide a data table into multiple partition tables based on the value range of preset column data in the data table stored in the source cluster; a sorting module, which is used to compare the order relationship between the values ​​of the preset column data in each partition table, obtain the migration priority of each partition table, and sort each partition table according to the migration priority of each partition table to obtain the sorting result of each partition table; a batch selection module, which is used to obtain the migration data volume corresponding to the current batch, and select the partition tables to be migrated in the current batch from the sorting results; wherein the total data volume of the selected partition tables is less than or equal to the migration data volume; and a migration module, which is used to migrate the data in the partition tables to be migrated in the current batch to the target cluster in sequence based on the migration priorities corresponding to the partition tables to be migrated in the current batch.

[0014] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned data migration method.

[0015] A fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned data migration method when the program instructions are executed by a processor.

[0016] The above scheme divides the data table into multiple partition tables based on the value range of the preset column data in the data table stored in the source cluster; compares the order relationship between the values ​​of the preset column data in each partition table to obtain the migration priority of each partition table, and sorts each partition table according to the migration priority of each partition table to obtain the sorting result of each partition table; obtains the migration data volume corresponding to the current batch, and selects the partition tables to be migrated in the current batch from the sorting results; wherein the total data volume of the selected partition tables is less than or equal to the migration data volume; based on the migration priorities corresponding to the partition tables to be migrated in the current batch, the data in the partition tables to be migrated in the current batch are migrated to the target cluster in sequence, so as to give priority to migrating high-priority data, ensure the effective operation of some businesses, reduce the impact of data migration on the business, and avoid migration failures caused by insufficient memory by migrating data in batches.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0019] Figure 1 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of data migration between clusters shown in an exemplary embodiment of the present application;

[0021] Figure 3 is a flow chart of a data migration method shown in an exemplary embodiment of the present application;

[0022] Figure 4 is a flow chart of a data migration method shown in another exemplary embodiment of the present application;

[0023] Figure 5 is a block diagram of a data migration device shown in an exemplary embodiment of the present application;

[0024] Figure 6 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application;

[0025] Figure 7 It is a schematic diagram of the structure of a computer-readable storage medium shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0026] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0027] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0028] The term "and / or" in this article is only an association information describing the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0029] The data migration method provided in the embodiment of the present application is described below.

[0030] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment may include a source cluster 110 and a target cluster 120, and the source cluster 110 and the target cluster 120 are connected to each other in communication.

[0031] The source cluster 110 and the target cluster 120 may be implemented based on a Greenplum database, a Postgresql database, a SQL Server database, or other databases, which is not limited in this application.

[0032] The source cluster 110 stores data in a data table, and can migrate all or part of the stored data to the target cluster 120 . The target cluster 120 receives and stores the data migrated from the source cluster 110 .

[0033] Taking Greenplum database as an example, the data migration process between source cluster 110 and target cluster 120 can be found in Figure 2 , Figure 2 A schematic diagram of data migration between clusters is shown as an exemplary embodiment of the present application. Figure 2 As shown, for each data to be transferred in the source cluster, perform the following steps: 1. Create a writable external table in the database of the source cluster; 2. Create a readable external table in the database of the target cluster; 3. Create a named pipe and the gpfdist process on the Segment hosts in the source cluster and the target cluster, where gpfdist is an external table access tool in the Greenplum database. The writable external table writes data from the source cluster database to a named pipe, and then the readable external table writes the data from the named pipe to the target cluster; 4. Execute an INSERT INTO SELECT statement in the database of the source cluster to insert the data in the database of the source cluster into the writable external table; 5. Execute an INSERT INTO SELECT statement in the database of the target cluster to insert the data from the readable external table into the target table; 6. Clear the external table, named pipe, and gpfdist process.

[0034] It should be noted that Figure 2 This is only an exemplary description. In actual application scenarios, if different cluster databases or different requirements are used, the data migration process will also be different, and this application does not limit this.

[0035] See also Figure 3 , Figure 3 is a flowchart of a data migration method shown in an exemplary embodiment of the present application. The data migration method can be applied to Figure 1 The implementation environment shown is specifically executed by the source cluster in the implementation environment.

[0036] like Figure 3 As shown, the data migration method at least includes steps S310 to S340, which are described in detail as follows:

[0037] Step S310: based on the value range of the preset column data in the data table stored in the source cluster, divide the data table into multiple partition tables.

[0038] In a database, a data table is the basic unit of data storage, and a field is a column in a data table. A field is used to store a specific type of data. The preset column data is all the data contained in the column corresponding to any field in the data table.

[0039] For example, a data table contains a time field for storing date information. The time field can be used as a preset column to obtain all data contained in the column corresponding to the time field to obtain the preset column data.

[0040] Of course, you can also select other types of fields as preset columns.

[0041] The preset columns may be pre-set based on experience, or may be flexibly determined based on actual conditions.

[0042] For example, determine the businesses that need to use the data table, record the read and write operations of each business on the data table, analyze the correlation between each field of the data table and the read and write frequency based on the read and write operations of each business on the data table, determine the fields that are strongly correlated with the read and write frequencies of each business based on the analysis results, and use the fields as preset columns.

[0043] For example, a data table is used to store image data, and the fields include but are not limited to the image type, storage path, upload account, upload date, image size, etc. One row of the data table corresponds to one image data, and each business performs read and write access to each image in the data table to perform related read and write operations. By analyzing the correlation between each field of the data table and the read and write frequency, it is found that the smaller the time interval between the upload date of the image data and the current time, the greater the read and write frequency, and the larger the time interval between the upload date of the image data and the current time, the smaller the read and write frequency. It can be seen that the field that is strongly correlated with the read and write frequency of each business is the upload date, and this field is used as the preset column.

[0044] For another example, a data table is used to store account information, and the fields include but are not limited to the account's unique identifier, account password, account level, device identifier of the account that has logged in, etc. One row of the data table corresponds to one account data, and each business performs read and write access to each account in the data table to perform related read and write operations. By analyzing the correlation between each field of the data table and the read and write frequency, it is found that the higher the account level of the account data, the greater the read and write frequency, and the lower the account level, the smaller the read and write frequency. It can be seen that the field that is strongly correlated with the read and write frequency of each business is the account level, and this field is used as the preset column.

[0045] After obtaining the preset columns that need to be referenced for data table partitioning, the data table is divided into multiple partition tables according to the value range of the preset column data in the data table stored in the source cluster. Among them, the partitioning strategies adopted by this application for data table partitioning include but are not limited to RANGE partitioning strategy, HASH partitioning strategy, LIST partitioning strategy and KEY partitioning strategy, etc., which can be flexibly selected according to the actual application situation, and this application does not limit this.

[0046] For example, the preset column data includes time data, and the value range includes a time range; based on the value range of the preset column data in the data table stored in the source cluster, the data table is divided into multiple partition tables, including: based on the time range of the time data in the data table stored in the source cluster, the data table is divided into multiple partition tables.

[0047] For example, data for the same day is written to the same partition table.

[0048] Of course, in addition to the above example of partitioning the data table from the time dimension, other fields can also be selected as preset columns that need to be referenced for data table partitioning, and this application will not provide detailed descriptions one by one.

[0049] Step S320: compare the order relationship between the values ​​of the preset column data in each partition table to obtain the migration priority of each partition table, and sort each partition table according to the migration priority of each partition table to obtain the sorting result of each partition table.

[0050] After the data table is divided according to the value range of the preset column data, the value range of the preset column data in different partition tables is different, and the migration priority of each partition table can be obtained based on the order relationship between the values ​​of the preset column data in each partition table.

[0051] It should be noted that the order relationship is different if the preset column data is different. For example, if the preset column data is time data, the order relationship is a time sequence relationship; if the preset column data is account level, the order relationship is a level size relationship.

[0052] For example, partition the data table from the time dimension, and the preset column data is date. The date corresponding to each row of data in partition table 1 is January 10, and the date corresponding to each row of data in partition table 2 is January 9. According to the time sequence between the dates in partition table 1 and partition table 2, the migration priority of partition table 1 is higher than that of partition table 2. By analogy, the migration priority of each partition table can be obtained by comparing the order relationship between the values ​​of the preset column data in each partition table.

[0053] Each partition table is sorted according to its migration priority to obtain a sorting result for each partition table.

[0054] Step S330: obtaining the migration data volume corresponding to the current batch, and selecting the partition tables to be migrated in the current batch from the sorting results; wherein the total data volume of the selected partition tables is less than or equal to the migration data volume.

[0055] It should be noted that the amount of migrated data can be used to limit the total amount of data contained in a partition table of a single migration, and can also be used to limit the total amount of partition tables of a single migration, which is not limited in this application.

[0056] The amount of migration data corresponding to the current batch may be preset based on experience, or may be flexibly calculated based on actual conditions.

[0057] For example, the amount of migration data corresponding to the current batch can be flexibly calculated based on available memory resources, data migration records corresponding to data migration in a historical time period, etc.

[0058] For example, the data migration record corresponding to the data migration performed within the historical time period is obtained. The data migration record records data such as the amount of migration data used for the data migration performed within the historical time period, the total amount of data that needs to be migrated, the time taken for migration, the number of failures, etc. These data are analyzed to score the data migration business within the historical time period. If the score is greater than a threshold, the difference between the total amount of data for the current data migration and the total amount of data for the historical data migration is calculated. Based on the difference, the amount of migration data used for the historical data migration is adjusted accordingly to obtain the amount of migration data corresponding to the current batch. Specifically, the larger the difference, the greater the increase in the amount of migration data used for the historical data migration, and the smaller the difference, the greater the decrease in the amount of migration data used for the historical data migration.

[0059] According to the migration data volume corresponding to the current batch, the partition table to be migrated in the current batch is selected from the sorting results.

[0060] It should be noted that when migrating multiple batches of data, the partition tables that have completed the migration need to be removed from the sorting results in real time. Also, when selecting the partition tables to be migrated in the current batch, the partition tables with high migration priority are selected from the sorting results first, until the total amount of data in the selected partition tables is less than or equal to the amount of data to be migrated.

[0061] Step S340: based on the migration priorities respectively corresponding to the partition tables to be migrated in the current batch, the data in the partition tables to be migrated in the current batch are migrated to the target cluster in sequence.

[0062] According to the migration priorities corresponding to the partition tables to be migrated in the current batch, the data in the partition tables are migrated in sequence, so as to migrate the data in the partition tables to the target cluster in sequence.

[0063] For example, see Figure 4 , Figure 4 A flowchart of a data migration method is shown in another exemplary embodiment of the present application. Figure 4 As shown, including:

[0064] Step S410: sorting according to the partition start value corresponding to the preset column data in each partition table.

[0065] For example, if partitioning is performed based on the time dimension, the preset column data is time data, and the partition start value can be the data row corresponding to the earliest or latest preset column data. The partition start values ​​corresponding to the preset column data in each partition table are sorted to obtain the sorting result of each partition table, such as the sorting result including part1, part2, part3, ..., part55 in sequence.

[0066] Step S420: grouping the sorting results according to the amount of migrated data.

[0067] The total amount of data in the partition table in each group is less than or equal to the amount of data to be migrated. For example, if the sorting results include part1, part2, part3, ..., part55 in sequence, and the amount of data to be migrated specifies that the number of partitions to be migrated each time is 10, then the group list obtained after grouping is: [part1, part10], [part11, part20], [part21, part30], [part31, part40], [part41, part50], [part51, part55].

[0068] Step S430: Determine whether the partition table in the grouping list has been processed. If it has been processed, the process ends. If it has not been processed, the process continues to loop from step S440 to step S460.

[0069] Step S440: Obtain the group corresponding to the current batch from the group list.

[0070] Each time a group in the group list is processed.

[0071] Step S450: taking the partition start value of the grouped start partition table as the start value of the migrated data, and taking the partition end value of the grouped end partition table as the end value of the migrated data, to obtain all partition fields between the start value of the migrated data and the end value of the migrated data.

[0072] Step S460: export the partition fields to a writable external table, so that the writable external table writes the data to the named pipe, and the readable external table writes the data to the target cluster from the named pipe.

[0073] The relevant database statements for obtaining all partition fields between the start value and the end value of the migration data and exporting the partition fields to a writable external table may include: insert into a writable external table select from data table where 1 = 1 and partition_key>= start value and partition_key< end value; Execute the above database statement in the source cluster to export the data of the current migration batch from the source cluster to a writable external table.

[0074] For example, a piece of data obtained from the grouping list is [part1, part10], where part1 is the start partition table of the migration and part10 is the end partition table of the migration. The partition start value of part1 is used as the start value of the migration data, and the partition end value of part10 is used as the end value of the migration data.

[0075] Next, some embodiments of the present application are described in detail.

[0076] In some implementations, step S310 divides the data table into multiple partition tables based on the value range of the preset column data in the data table stored in the source cluster, including steps S311 to S313.

[0077] Step S311: Obtain the range granularity for dividing the data table, where the range granularity is used to limit the range size of the value range.

[0078] For example, if the preset column data is data corresponding to a time field, the value range is used to limit the time range, and the range granularity is used to limit the range size of the time range. The larger the range granularity, the larger the time range, and the smaller the range granularity, the smaller the time range.

[0079] Step S312: determining a partition start value and a partition end value of the value range based on the range granularity;

[0080] Taking the time field as an example, if the range granularity is 24 hours, the partition start value and partition end value of each value range obtained by division include 0:00 on January 1 to 24:00 on January 1, 0:00 on January 2 to 24:00 on January 2, ..., and so on; if the range granularity is 12 hours, the partition start value and partition end value of each value range obtained by division include 0:00 on January 1 to 12:00 on January 1, 12:00 on January 1 to 12:00 on January 1, ..., and so on.

[0081] Step S313: divide the data rows in the data table whose preset column data is within the partition start value and the partition end value into the partition table corresponding to the value range.

[0082] In one implementation, obtaining the range granularity for dividing the data table in step S311 includes steps S3111 to S3112.

[0083] Step S3111: Obtain data read and write operations for data table records.

[0084] If a data read or write operation on a data table is detected, relevant information of the data read or write operation is recorded, such as the business initiated by the read or write operation, the data content involved in the read or write operation, and the like.

[0085] Step S3112: setting the range granularity for dividing the data table based on data read and write operations.

[0086] Analyze data read and write operations, and set the range granularity for dividing the data table based on the analysis results.

[0087] Exemplarily, setting the range granularity for dividing the data table based on data read and write operations includes: counting the number of data read and write operations within a preset time period to obtain a statistical result; and calculating the range granularity for dividing the data table based on the statistical result.

[0088] For example, the number of data read and write operations within a preset time period is inversely proportional to the range granularity. That is, the statistical results show that the higher the number of data read and write operations within the preset time period, the smaller the range granularity of the data table division is, so as to reduce the amount of data contained in a single partition table. When the partition tables are subsequently migrated, the amount of data that cannot be used in time by the business is reduced to ensure the business execution effect. Also, the statistical results show that the lower the number of data read and write operations within the preset time period, the larger the range granularity of the data table division is, so as to quickly migrate infrequently used data and improve the migration efficiency of infrequently used data.

[0089] Of course, different calculation functions can be set for different data tables. The calculation function is used to limit the correlation between the number of data read and write operations within a preset time period and the range granularity. The range granularity is obtained by substituting the number of data read and write operations within the preset time period into the calculation function corresponding to the data table.

[0090] Exemplarily, the range granularity for dividing the data table based on the data read and write operation setting includes: obtaining the fields and services associated with each data read and write operation, and statistically obtaining the services associated with each field; and obtaining the range granularity matching the data table based on the importance corresponding to the services associated with each field.

[0091] For example, the highest importance of the associated business can be obtained to determine the range granularity that matches the importance; or, the importance corresponding to the business associated with each field can be weighted and summed to obtain the final importance, and the range granularity that matches the importance can be determined. The weight used for weighting can be preset or determined based on the number of data read and write operations corresponding to each field. The higher the number of data read and write operations, the greater the weight corresponding to the field. Optionally, the importance and range granularity are inversely correlated, that is, the higher the importance, the smaller the range granularity, and the lower the importance, the larger the range granularity, so as to ensure the amount of data that can be used by important businesses during data migration.

[0092] For example, the fields contained in the data table include account number, password, level, login device, etc. By analyzing the read and write operations of the recorded data table, it is found that the services associated with "account number" and "password" include registration service and login service, the services associated with "level" include home page interface display, and the services associated with "login device" include security information query service; the importance of each of the above-mentioned associated services is obtained to determine the range granularity that matches the data table according to the importance of each associated business.

[0093] In some implementations, obtaining the migration data volume corresponding to the current batch in step S330 includes:

[0094] Step S331: Detect available memory resources of the source cluster and / or the target cluster;

[0095] Step S332: Based on available memory resources, set the amount of data to be migrated at one time.

[0096] For example, the larger the available memory resources of the source cluster and / or the target cluster, the larger the amount of data to be migrated at a time; the smaller the available memory resources of the source cluster and / or the target cluster, the smaller the amount of data to be migrated at a time, thereby reducing the possibility of data migration failure due to insufficient memory and improving the success rate of data migration.

[0097] In some implementations, detecting available memory resources of the source cluster and / or the target cluster in step S331 includes:

[0098] Step S3311: Record the memory resource usage of the source cluster and / or the target cluster in the historical time period;

[0099] Step S3312: Analyze the memory resource change trend of the source cluster and / or the target cluster in the future time period based on the memory resource usage in the historical time period;

[0100] Step S3313: Based on the memory resource change trend, the available memory resources of the source cluster and / or the target cluster are calculated.

[0101] The memory resource usage of the source cluster and / or target cluster in the historical time period is detected and recorded to analyze the fluctuation pattern of the memory resource usage in the historical time period, and then combined with the analyzed fluctuation pattern of the memory resource usage to predict the memory resource change trend of the source cluster and / or target cluster in the future time period, so as to obtain the available memory resources of the source cluster and / or target cluster in the future time period, and use them as the available memory resources of the source cluster and / or target cluster.

[0102] Through the above embodiments, available memory resources of the source cluster and / or the target cluster in the subsequent data migration process can be predicted, further ensuring that data can be successfully migrated and improving data migration efficiency.

[0103] The data migration method provided by the present application divides a data table into multiple partition tables based on the value range of preset column data in the data table stored in the source cluster; compares the order relationship between the values ​​of the preset column data in each partition table to obtain the migration priority of each partition table, and sorts each partition table according to the migration priority of each partition table to obtain the sorting result of each partition table; obtains the migration data volume corresponding to the current batch, and selects the partition tables to be migrated in the current batch from the sorting results; wherein the total data volume of the selected partition tables is less than or equal to the migration data volume; based on the migration priorities corresponding to the partition tables to be migrated in the current batch, the data in the partition tables to be migrated in the current batch are migrated to the target cluster in sequence, so as to give priority to migrating data with high priority, ensure the effective operation of some businesses, reduce the impact of data migration on businesses, and avoid migration failures caused by insufficient memory by migrating data in batches.

[0104] Figure 5 FIG. 1 is a block diagram of a data migration device shown in an exemplary embodiment of the present application. Figure 5 As shown, the exemplary data migration device 500 includes: a table partitioning module 510, a sorting module 520, a batch selection module 530 and a migration module 540. Specifically:

[0105] A table partitioning module 510 is used to divide the data table into multiple partition tables based on the value range of the preset column data in the data table stored in the source cluster;

[0106] A sorting module 520 is used to compare the order relationship between the values ​​of the preset column data in each partition table to obtain the migration priority of each partition table, and sort each partition table according to the migration priority of each partition table to obtain the sorting result of each partition table;

[0107] The batch selection module 530 is used to obtain the migration data volume corresponding to the current batch, and select the partition tables to be migrated in the current batch from the sorting results; wherein the total data volume of the selected partition tables is less than or equal to the migration data volume;

[0108] The migration module 540 is used to migrate the data in the partition tables to be migrated in the current batch to the target cluster in sequence based on the migration priorities respectively corresponding to the partition tables to be migrated in the current batch.

[0109] It should be noted that the data migration device provided in the above embodiment and the data migration method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In actual application, the data migration device provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0110] See also Figure 6 , Figure 6 6 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 600 includes a memory 601 and a processor 602, and the processor 602 is used to execute program instructions stored in the memory 601 to implement the steps in any of the above data migration method embodiments. In a specific implementation scenario, the electronic device 600 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 600 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0111] Specifically, the processor 602 is used to control itself and the memory 601 to implement the steps in any of the above data migration method embodiments. The processor 602 can also be called a central processing unit (CPU). The processor 602 may be an integrated circuit chip with signal processing capabilities. The processor 602 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 602 can be implemented by an integrated circuit chip.

[0112] See also Figure 7 , Figure 7 The computer-readable storage medium 700 stores program instructions 710 that can be executed by a processor, and the program instructions 710 are used to implement the steps in any of the above data migration method embodiments.

[0113] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0114] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0115] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0116] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

Claims

1. A data migration method, characterized in that: Applied to a source cluster, the source cluster stores data through a data table, and the method includes: Obtaining data read and write operations for data table records, obtaining statistics of the businesses associated with each field based on the fields and businesses associated with the data read and write operations, and setting a range granularity for dividing the data table based on the importance of the businesses associated with each field, wherein the range granularity is used to limit the range size of a value range; Determining a partition start value and a partition end value of the value range based on the range granularity; The data rows whose preset column data in the data table is within the partition start value and the partition end value are divided into the partition table corresponding to the value range, so as to divide the data table into a plurality of partition tables; Compare the order relationship between the values ​​of the preset column data in each partition table to obtain the migration priority of each partition table, and sort each partition table according to the migration priority of each partition table to obtain the sorting result of each partition table; Obtaining the migration data volume corresponding to the current batch, and selecting the partition tables to be migrated in the current batch from the sorting results; wherein the total data volume of the selected partition tables is less than or equal to the migration data volume; Based on the migration priorities respectively corresponding to the partition tables to be migrated in the current batch, the data in the partition tables to be migrated in the current batch are migrated to the target cluster in sequence.

2. The method according to claim 1, characterized in that: The method further comprises: Counting the number of data read and write operations within a preset time period to obtain statistical results; Based on the statistical results, a range granularity for dividing the data table is calculated.

3. The method according to claim 1, characterized in that The preset column data includes time data, and the value range includes a time range; the method further includes: Based on the time range of the time data in the data table stored in the source cluster, the data table is divided into multiple partition tables.

4. The method according to claim 1, characterized in that: The step of obtaining the migration data amount corresponding to the current batch includes: Detecting available memory resources of the source cluster and / or the target cluster; The amount of data to be migrated at a time is set based on the available memory resources.

5. The method according to claim 4, characterized in that The detecting available memory resources of the source cluster and / or the target cluster includes: Recording memory resource usage of the source cluster and / or the target cluster within a historical period; Based on the memory resource usage in the historical time period, analyzing the memory resource change trend of the source cluster and / or the target cluster in a future time period; Based on the memory resource change trend, available memory resources of the source cluster and / or the target cluster are calculated.

6. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the steps in the method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the steps in the method according to any one of claims 1 to 5.

Images