Partition-based data migration method and apparatus

Abstract

The present disclosure provides a partition-based data migration method, which can be applied to the field of financial technology. The method is applied to a multi-cluster distributed system, and includes: determining to-be-migrated data information and a target migration partition according to a data migration configuration file; performing validity verification on the target migration partition; when it is determined that the target migration partition is valid, obtaining a data migration direction, the data migration direction representing a data migration direction between multiple clusters; executing a migrated data acquisition script according to the data migration direction, to obtain to-be-migrated data; and executing a data migration transmission script according to the to-be-migrated data, the target migration partition and the data migration direction, to perform data migration at a partition granularity. The present disclosure also provides a partition-based data migration device, equipment, storage medium and program product.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, specifically to the field of automated data migration technology, and more specifically to a partition-based data migration method, apparatus, device, storage medium, and program product. Background Art

[0002] In big data technology, distributed ecosystems, such as Hadoop, have been widely used. During big data system testing, due to factors such as test effectiveness and cost, test environments often lack the abundant cluster resources of production environments, resulting in limited data storage. Inconsistent testing cycles across test environments lead to inconsistent data storage across them. However, the Hive architecture used for data warehouse management does not support single-record data modifications.

[0003] In the related art, there is a method of migrating the entire Hive table. This data migration method contains a large amount of invalid data, resulting in reduced data migration efficiency.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0005] In view of the above problems, the present disclosure provides a partition-based data migration method, apparatus, device, medium, and program product for improving data migration efficiency.

[0006] According to a first aspect of the present disclosure, a partition-based data migration method is provided, which is applied to a multi-cluster distributed system. The data migration method includes:

[0007] Determine the data to be migrated and the target migration partition based on the data migration configuration file;

[0008] Performing validity verification on the target migration partition;

[0009] When it is determined that the target migration partition is valid, obtaining a data migration direction, where the data migration direction represents a data migration direction between multiple clusters;

[0010] Executing a migration data acquisition script according to the data migration direction to acquire data to be migrated; and

[0011] A data migration transmission script is executed according to the data to be migrated, the target migration partition, and the data migration direction to perform data migration at a partition granularity.

[0012] According to an embodiment of the present disclosure, executing a migration data acquisition script according to the data migration direction to acquire the data to be migrated includes:

[0013] Generate a migration data acquisition script according to the data information to be migrated;

[0014] Verify the cluster where the partition to be migrated is located according to the data migration direction;

[0015] Save the data to be migrated locally according to the partition granularity.

[0016] According to an embodiment of the present disclosure, generating a migration data acquisition script according to the to-be-migrated data information includes:

[0017] Traversing the data information to be migrated to determine a target input partition;

[0018] generating a first target file according to the target input partition; and

[0019] Generate a migration data acquisition script according to the partition range recorded in the first target file.

[0020] According to an embodiment of the present disclosure, the data information to be migrated includes the name of the database to be migrated, the name of the data table to be migrated, and the name of the partition to be migrated. Traversing the data information to be migrated to determine the target input partition includes:

[0021] Traverse the partition names to be migrated to determine the available range of partitions to be migrated;

[0022] A target input partition is determined according to the available range of partitions to be migrated.

[0023] According to an embodiment of the present disclosure, executing a data migration transfer script according to the data to be migrated, the target migration partition, and the data migration direction to perform partition-granular data migration includes:

[0024] generating a data migration transfer script according to the data to be migrated and the target migration partition;

[0025] The data migration transmission script is executed according to the data migration direction to perform data migration at the partition granularity.

[0026] According to an embodiment of the present disclosure, generating a data migration transmission script according to the data to be migrated and the target migration partition includes:

[0027] Traversing the acquired data directory according to the migration data to generate a second target file;

[0028] Generate a data migration and transmission script based on the data of the second target file.

[0029] According to an embodiment of the present disclosure, before executing the migration data transmission script, the method further includes:

[0030] Generate a corresponding hive partition script according to the second target file;

[0031] Execute the hive partition script to create hive partitions.

[0032] According to an embodiment of the present disclosure, the performing validity check on the target migration partition includes:

[0033] Determine a target cluster according to the target migration partition;

[0034] When it is determined that the target migration partition exists in the target cluster, determining that the target migration partition is invalid; and

[0035] When it is determined that the target migration partition does not exist in the target cluster, the target migration partition is determined to be valid.

[0036] A second aspect of the present disclosure provides a partition-based data migration device, comprising: a first determination module, configured to determine data information to be migrated and a target migration partition according to a data migration configuration file;

[0037] A verification module, configured to verify the validity of the target migration partition;

[0038] A first acquisition module is configured to acquire a data migration direction when it is determined that the target migration partition is valid, where the data migration direction represents a data migration direction between multiple clusters;

[0039] A second acquisition module is used to execute a migration data acquisition script according to the data migration direction to obtain the data to be migrated;

[0040] The transmission module is configured to execute a data migration transmission script according to the data to be migrated, the target migration partition, and the data migration direction, so as to perform data migration at a partition granularity.

[0041] According to an embodiment of the present disclosure, the second acquisition module includes:

[0042] A first generating submodule is configured to generate a migration data acquisition script based on the data to be migrated;

[0043] An authentication submodule, configured to authenticate the cluster where the partition to be migrated is located according to the data migration direction;

[0044] The data to be migrated storage submodule is used to save the data to be migrated locally according to the partition granularity.

[0045] According to an embodiment of the present disclosure, the first generation submodule includes:

[0046] A traversal unit, configured to traverse the data information to be migrated to determine a target input partition;

[0047] A first generating unit is configured to generate a first target file according to the target input partition; and

[0048] The second generating unit is configured to generate a migration data acquisition script according to the partition range recorded in the first target file.

[0049] According to an embodiment of the present disclosure, the traversal unit includes:

[0050] A traversal subunit, configured to traverse the names of the partitions to be migrated and determine a range of available partitions to be migrated;

[0051] The target input partition determination subunit is configured to determine the target input partition according to the available range of partitions to be migrated.

[0052] According to an embodiment of the present disclosure, the transmission module includes:

[0053] A second generating submodule is configured to generate a data migration transmission script according to the data to be migrated and the target migration partition;

[0054] The migration submodule is used to execute the data migration transmission script according to the data migration direction to perform data migration at the partition granularity.

[0055] According to an embodiment of the present disclosure, the second generation submodule includes:

[0056] A third generating unit is configured to traverse the acquired data directory according to the migration data and generate a second target file;

[0057] The fourth generating unit is configured to generate a data migration transmission script according to the data of the second target file.

[0058] According to an embodiment of the present disclosure, it further includes:

[0059] A generation module, configured to generate a corresponding hive partition script according to the second target file;

[0060] The execution module is used to execute the hive partition script to establish hive partitions.

[0061] The third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned partition-based data migration method.

[0062] A fourth aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned partition-based data migration method.

[0063] A fifth aspect of the present disclosure further provides a computer program product, including a computer program, which implements the above-mentioned partition-based data migration method when executed by a processor.

[0064] A partition-based data migration method provided by an embodiment of the present disclosure determines the data information to be migrated and the target migration partition according to a data migration configuration file. After the validity of the target migration partition is verified, when it is determined that the target migration partition is valid, the data migration direction is obtained, a migration data acquisition script and a data migration transmission script are generated, and the scripts are executed to complete the data migration at the partition granularity. Through the method of the present disclosure, data migration operations based on partition granularity are realized within the same cluster, across clusters, within the same table, and across tables, thereby improving data utilization and data migration efficiency, thereby improving testing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0066] Figure 1 Schematically illustrates an application scenario diagram of a partition-based data migration method, apparatus, device, medium, and program product according to an embodiment of the present disclosure;

[0067] Figure 2 The following schematically shows a flow chart of a partition-based data migration method according to an embodiment of the present disclosure;

[0068] Figure 3 A flowchart of a method for obtaining data to be migrated according to an embodiment of the present disclosure is schematically shown;

[0069] Figure 4 Schematically shows a flow chart of a method for generating a migration data acquisition script according to an embodiment of the present disclosure;

[0070] Figure 5 The following schematically shows a flow chart of a data migration method according to an embodiment of the present disclosure;

[0071] Figure 6 The flowchart schematically shows a method for generating a data migration transmission script according to an embodiment of the present disclosure;

[0072] Figure 7 The following schematically shows a flow chart of a method for establishing a hive partition according to an embodiment of the present disclosure;

[0073] Figure 8 The following schematically shows a structural block diagram of a partition-based data migration device according to an embodiment of the present disclosure;

[0074] Figure 9 A block diagram of an electronic device suitable for implementing a partition-based data migration method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0075] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0076] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0077] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0078] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0079] First, the terms that appear in this disclosure are explained:

[0080] Hadoop is a distributed systems infrastructure and a software framework that enables distributed processing of large amounts of data in a reliable, efficient, and scalable manner. At the bottom of Hadoop is HDFS, which stores files across all storage nodes in the Hadoop cluster. Above HDFS is the MapReduce engine.

[0081] HDFS: Hadoop Distributed File System, HDFS distributed file system is a Hadoop distributed file system with high fault tolerance. It can be deployed on low-cost hardware and is widely used for big data file storage.

[0082] HIVE: Hadoop-based data warehouse software that can query and manage petabyte-scale distributed data. Hive partitions correspond to HDFS directories, creating a mapping relationship. Therefore, in this example, new HIVE partitions must be created to map them to HDFS for queries. Hive also doesn't support modifying individual fields, so migration is required.

[0083] In big data technology, distributed ecosystems, such as Hadoop, have been widely used. During big data system testing, due to factors like test effectiveness and cost, test environments often lack the abundant cluster resources of production environments, resulting in limited data storage. The inconsistent testing cycles across test environments lead to inconsistent data storage. Furthermore, the test dates in test and production environments also differ. Consequently, when loading big data using Kafka data transmission, the test date and the actual date may be inconsistent. This significantly reduces testing efficiency and data utilization.

[0084] Based on the above technical problems, an embodiment of the present disclosure provides a partition-based data migration method, which is applied to a multi-cluster distributed system. The method includes: determining the data information to be migrated and the target migration partition according to a data migration configuration file; performing validity verification on the target migration partition; when it is determined that the target migration partition is valid, obtaining the data migration direction, which represents the data migration direction between multiple clusters; executing a migration data acquisition script according to the data migration direction to obtain the data to be migrated; and executing a data migration transmission script according to the data to be migrated, the target migration partition and the data migration direction to perform data migration at a partition granularity.

[0085] Figure 1 The application scenario diagram of the partition-based data migration method, apparatus, device, medium and program product according to an embodiment of the present disclosure is schematically shown.

[0086] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a distributed data migration scenario. A network 104 is used to provide a medium for communication links between terminal devices 101, 102, 103 and server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0087] Testers can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0088] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0089] Server 105 can be a data migration server, which verifies the data to be migrated and the target migration partition based on the data migration configuration file input by the tester. For example, in response to the data migration instruction issued by the tester using the terminal devices 101, 102, and 103, the data migration server can generate a migration data acquisition script and a data migration transmission script for the received data migration configuration file, and execute the script to perform data migration at the partition granularity.

[0090] It should be noted that the partition-based data migration method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the partition-based data migration device provided in the embodiment of the present disclosure can generally be set in the server 105. The partition-based data migration method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the partition-based data migration device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0091] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0092] It should be noted that the partition-based data migration method and device determined in the embodiments of the present disclosure can be used in the field of financial technology, and can also be used in any field other than the financial field. The application field of the partition-based data migration method and device determined in the embodiments of the present disclosure is not limited.

[0093] The following will be based on Figure 1 The scene described by Figures 2 to 7 The partition-based data migration method according to an embodiment of the present disclosure is described in detail.

[0094] Figure 2 The flowchart of the partition-based data migration method according to an embodiment of the present disclosure is schematically shown. Figure 2 As shown, the partition-based data migration method of this embodiment includes operations S210 to S250, which can be executed by a server or other computing devices. The embodiment of the present disclosure performs partition-granular data migration from the HDFS level.

[0095] In operation S210 , data information to be migrated and a target migration partition are determined according to the data migration configuration file.

[0096] According to an embodiment of the present disclosure, the data information to be migrated includes the name of the database to be migrated, the name of the data table to be migrated, and the name of the partition to be migrated.

[0097] In one example, a user generates a data migration configuration file based on their own testing requirements to determine the data information to be migrated and the target migration partition, where the data information to be migrated includes the name of the database to be migrated, the name of the data table to be migrated, and the name of the partition to be migrated.

[0098] In operation S220, the validity of the target migration partition is checked.

[0099] In one example, in order to improve the efficiency of data migration, the target migration partition is first checked for validity to verify the availability of the target migration partition. Scalable, the target cluster is determined based on the target migration partition; when it is determined that the target migration partition exists in the target cluster, the target migration partition is determined to be invalid. Determine whether the corresponding partition exists in the target cluster. If data exists in the corresponding partition of the target cluster, the entire process is interrupted. (For example, through Hadoop fs-ls / user / hive / ecs / tb_1010, all partitions under the corresponding table are obtained; and correspond to the migration data range submitted by the user. If there is already a corresponding partition under the target cluster, the entire process is interrupted, and the user is prompted that the target cluster already has data. If there is no corresponding partition, proceed to the next step. When it is determined that the target migration partition does not exist in the target cluster, the target migration partition is determined to be valid.

[0100] In addition, the data to be migrated must be verified. This is primarily used to determine whether the user has permission to migrate the table name they entered. Permission is determined by the beginning of the table name. For example, a table name beginning with DCM_ or FCM_ indicates permission. Specifically, based on the user's selected environment, entered library name, and table name, a corresponding partition size query statement (for example, hadoop fs -du-sh / user / hive / ecs / tb_1010) is generated. The user then authenticates to the corresponding Hadoop cluster and executes the partition size query statement. The query results are fed back to the frontend. If the query succeeds, the table name entered by the user is confirmed to be correct. If the query fails, feedback is sent to the frontend indicating that the table name is incorrect.

[0101] In operation S230 , when it is determined that the target migration partition is valid, a data migration direction is acquired.

[0102] According to an embodiment of the present disclosure, the data migration direction represents the data migration direction between multiple clusters.

[0103] In one example, when performing data migration, it is necessary to determine the direction of migration. Taking the two clusters AB as an example, it is necessary to determine whether to migrate from A to B or from B to A. In the embodiment of the present disclosure, the data migration involving three clusters includes a total of 6 migration directions; according to the different buttons selected by the user in the foreground, the background uses different identifiers to determine the corresponding migration direction. Since hive supports different types of retrieval, such as date, time, etc., they can all be used as partition fields. Then in the actual test environment, multiple partition types may be included. Therefore, it is also necessary to determine the migration type. In the embodiment of the present disclosure, it mainly includes four different partition types and corresponding migration types: 2022-04-27, 2022-04-27-ar, 2022-04-27-051, and 2022-04-27-01.

[0104] In operation S240 , a migration data acquisition script is executed according to the data migration direction to acquire data to be migrated.

[0105] In an example, first generate a migration data acquisition script based on the data migration direction. The specific process of the migration data acquisition script can be found in Figure 4 The generated migration data acquisition script is executed to obtain the data to be migrated from the data partition to be migrated to the local computer. For the specific process, please refer to Figure 3 Operations S241 to S243 are shown.

[0106] In operation S250 , a data migration transmission script is executed according to the data to be migrated, the target migration partition, and the data migration direction to perform partition-granular data migration.

[0107] In an example, first generate the data migration transfer script. For the specific process, see Figure 6 Operation S2511 and operation S2512 are shown. According to the target migration partition, it is authenticated to the corresponding target cluster, and the corresponding data migration transfer script is executed to complete the partition-granularity data migration.

[0108] A partition-based data migration method provided by an embodiment of the present disclosure determines the data information to be migrated and the target migration partition according to a data migration configuration file. After the validity of the target migration partition is verified, when it is determined that the target migration partition is valid, the data migration direction is obtained, a migration data acquisition script and a data migration transmission script are generated, and the scripts are executed to complete the data migration at the partition granularity. Through the method of the present disclosure, data migration operations based on partition granularity are realized within the same cluster, across clusters, within the same table, and across tables, thereby improving data utilization and data migration efficiency, thereby improving testing efficiency.

[0109] Figure 3 The flowchart of the method for obtaining data to be migrated according to an embodiment of the present disclosure is schematically shown. Figure 4 The flowchart of the migration data acquisition script generation method according to an embodiment of the present disclosure is schematically shown.

[0110] like Figure 3 As shown, operation S240 includes operations S241 to S243.

[0111] In operation S241 , a migration data acquisition script is generated according to the information of the data to be migrated.

[0112] In operation S242 , the cluster where the partition to be migrated is located is authenticated according to the data migration direction.

[0113] In operation S243 , the data to be migrated is saved locally according to the partition granularity.

[0114] In one example, a migration data acquisition script is generated based on the data to be migrated to automatically acquire the source data. For details, see Figure 4 Operations S2411 to S2413 are shown.

[0115] In one example, after generating a migration data acquisition script, the original cluster where the partition to be migrated is located is authenticated according to the data migration direction, and the data to be migrated is saved locally according to the partition granularity.

[0116] like Figure 4 As shown, operation S240 includes operations S2411 to S2413.

[0117] In operation S2411 , the to-be-migrated data information is traversed to determine a target input partition.

[0118] According to an embodiment of the present disclosure, the names of the partitions to be migrated are traversed to determine an available range of partitions to be migrated; and a target input partition is determined based on the available range of partitions to be migrated.

[0119] In operation S2412 , a first target file is generated according to the target input partition.

[0120] In operation S2413, a migration data acquisition script is generated according to the partition range of the first target file record.

[0121] In one example, to facilitate batch data migration, users can enter a range of partitions in the data migration configuration file during the data input phase to migrate multiple partitions. During the actual migration process, the entered partition range is traversed to obtain the partitions that match the input as the target input partitions. This partition is recorded in the time.txt file, which serves as the first target file. Based on the partition range recorded in time.txt, the corresponding migration data acquisition script is generated. For example, the command "hadoop fs -get / user / hive / ecs.db / dcm_ecs_tb1010_s / " is used.

[0122] After the data to be migrated is acquired, it is necessary to migrate the data to be migrated. Figure 5 The flowchart of the data migration method according to the embodiment of the present disclosure is schematically shown. Figure 6 The flowchart of the method for generating a data migration transmission script according to an embodiment of the present disclosure is schematically shown. Figure 5 As shown, operation S250 includes operations S251 to S252.

[0123] In operation S251 , a data migration transmission script is generated according to the data to be migrated and the target migration partition.

[0124] In operation S252 , a data migration transmission script is executed according to the data migration direction to perform data migration at a partition granularity.

[0125] In an example, similarly, first generate a data migration script based on the data to be migrated and the target migration partition. Figure 6 Operation S2511 and operation S2512 are shown. Executing the data migration transfer script according to the data migration direction can achieve data migration at the partition granularity.

[0126] like Figure 6 As shown, operation S251 includes operation S2511 and operation S2512.

[0127] In operation S2511, the acquired data directory is traversed according to the data to be migrated to generate a second target file. In operation S2512, a data migration transmission script is generated according to the data in the second target file.

[0128] In one example, a user submitted data for a certain time range, but some partitions lacked data. For example, if a user submitted data for the range 03:01-03:31, the original cluster might not have data for 03:04. To reduce the migration of invalid partitions and improve migration efficiency, in actual use, a corresponding data transfer script is generated based on the data obtained from the original cluster. First, the directory of the obtained data is traversed and recorded in the data.txt file, which serves as the second target file. For example, based on the locally obtained data to be migrated, the directory is traversed to obtain data.txt. Based on the data recorded in data.txt, the corresponding migration data transfer script is generated. For example, hadoop fs -put / user / hive / ecs.db / dcm_ecs_tb1010s / .

[0129] Figure 7 The flowchart of the method for establishing a hive partition according to an embodiment of the present disclosure is schematically shown. Before executing the data migration transfer script to complete the data migration, in order to enable normal use by business personnel, a hive partition corresponding to the HDFS needs to be established, including operations S310 and S320.

[0130] In operation S310, a corresponding hive partitioning script is generated according to the second target file.

[0131] In operation S320, the hive partitioning script is executed to create a hive partition.

[0132] In one example, a Hive partitioning script is generated based on the data.txt file. For example, alter table ecs,tab_10010s add partition(pt_dt='AAA') is used. This Hive partitioning script is then run to create Hive partitions.

[0133] The disclosed embodiments provide a partition-based data migration method that can help users implement cross-cluster, same-cluster, and same-table data migration operations faster and more efficiently. It also supports migration of regular dates, multiple batches, special partition fields, and Hive partitions, thereby improving data utilization, reducing business data preparation time, and improving testing efficiency.

[0134] Based on the above partition-based data migration method, the present disclosure also provides a partition-based data migration device. Figure 8The device is described in detail.

[0135] Figure 8 The structure block diagram of the partition-based data migration device according to an embodiment of the present disclosure is schematically shown.

[0136] like Figure 8 As shown, the partition-based data migration device 800 of this embodiment includes a first determination module 810 , a verification module 820 , a first acquisition module 830 , a second acquisition module 840 and a transmission module 850 .

[0137] The first determining module 810 is used to determine the data information to be migrated and the target migration partition according to the data migration configuration file. In one embodiment, the first determining module 810 can be used to perform the operation S210 described above, which will not be repeated here.

[0138] The verification module 820 is used to verify the validity of the target migration partition. In one embodiment, the verification module 820 can be used to perform the operation S220 described above, which will not be described in detail here.

[0139] The first acquisition module 830 is used to acquire a data migration direction when determining that the target migration partition is valid, wherein the data migration direction represents the data migration direction between multiple clusters. In one embodiment, the first acquisition module 830 can be used to perform the operation S230 described above, which will not be repeated here.

[0140] The second acquisition module 840 is used to execute the migration data acquisition script according to the data migration direction to obtain the data to be migrated. In one embodiment, the output module 840 can be used to perform the operation S240 described above, which will not be repeated here.

[0141] The transmission module 850 is used to execute the data migration transmission script according to the data to be migrated, the target migration partition and the data migration direction to perform partition-level data migration. In one embodiment, the transmission module 850 can be used to perform the operation S250 described above, which will not be repeated here.

[0142] According to an embodiment of the present disclosure, the second acquisition module 840 includes: a first generation submodule, an authentication submodule, and a to-be-migrated data storage submodule.

[0143] The first generation submodule is used to generate a migration data acquisition script based on the to-be-migrated data information. In one embodiment, the first generation submodule can be used to perform the operation S241 described above, which will not be described in detail here.

[0144] The authentication submodule is used to authenticate the cluster where the partition to be migrated is located according to the data migration direction. In one embodiment, the authentication submodule can be used to perform the operation S242 described above, which will not be repeated here.

[0145] The data to be migrated storage submodule is used to store the data to be migrated locally according to the partition granularity. In one embodiment, the data to be migrated storage submodule can be used to perform the operation S243 described above, which will not be repeated here.

[0146] According to an embodiment of the present disclosure, the first generation submodule includes: a traversal unit, a first generation unit and a second generation unit.

[0147] The traversal unit is used to traverse the data information to be migrated to determine the target input partition. In one embodiment, the traversal unit can be used to perform the operation S2411 described above, which will not be repeated here.

[0148] The first generating unit is configured to generate a first target file according to the target input partition. In one embodiment, the first generating unit may be configured to perform the operation S2412 described above, which will not be described in detail here.

[0149] The second generating unit is used to generate a migration data acquisition script according to the partition range of the first target file record. In one embodiment, the second generating unit can be used to perform the operation S2413 described above, which will not be repeated here.

[0150] According to an embodiment of the present disclosure, the traversal unit includes: a traversal subunit and a target input partition determination subunit.

[0151] The traversal subunit is used to traverse the partition names to be migrated and determine the available range of partitions to be migrated. In one embodiment, the traversal subunit can be used to perform the operation S2411 described above, which will not be repeated here.

[0152] The target input partition determination subunit is configured to determine the target input partition according to the available range of partitions to be migrated. In one embodiment, the target input partition determination subunit may be configured to perform the operation S2411 described above, which will not be described in detail here.

[0153] According to an embodiment of the present disclosure, the transmission module 850 includes: a second generation submodule and a migration submodule.

[0154] The second generation submodule is used to generate a data migration transmission script according to the data to be migrated and the target migration partition. In one embodiment, the second generation submodule can be used to perform the operation S251 described above, which will not be repeated here.

[0155] The migration submodule is used to execute the data migration transmission script according to the data migration direction to perform partition-level data migration. In one embodiment, the migration submodule can be used to perform the operation S252 described above, which will not be repeated here.

[0156] According to an embodiment of the present disclosure, the second generating submodule includes: a third generating unit and a fourth generating unit.

[0157] The third generating unit is used to traverse the acquired data directory according to the migration data to generate a second target file. In one embodiment, the third generating unit can be used to perform the operation S2511 described above, which will not be repeated here.

[0158] The fourth generating unit is configured to generate a data migration transmission script based on the data of the second target file. In one embodiment, the fourth generating unit may be configured to execute the operation S2512 described above, which will not be described in detail here.

[0159] According to an embodiment of the present disclosure, it further includes: a generating module and an executing module.

[0160] The generation module is used to generate a corresponding hive partition script according to the second target file. In one embodiment, the generation module can be used to perform the operation S310 described above, which will not be repeated here.

[0161] The execution module is used to execute the hive partitioning script to establish the hive partition. In one embodiment, the execution module can be used to perform the operation S320 described above, which will not be repeated here.

[0162] According to an embodiment of the present disclosure, any multiple modules among the first determination module 810, the verification module 820, the first acquisition module 830, the second acquisition module 840, and the transmission module 850 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first determination module 810, the verification module 820, the first acquisition module 830, the second acquisition module 840, and the transmission module 850 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the first determination module 810 , the verification module 820 , the first acquisition module 830 , the second acquisition module 840 and the transmission module 850 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0163] Figure 9 A block diagram of an electronic device suitable for implementing a partition-based data migration method according to an embodiment of the present disclosure is schematically shown.

[0164] like Figure 9 As shown, the electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. The processor 901 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0165] Various programs and data required for the operation of the electronic device 900 are stored in the RAM 903. The processor 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. The processor 901 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 902 and / or the RAM 903. It should be noted that the programs may also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0166] According to an embodiment of the present disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the I / O interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card or a modem. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 910 as needed, so that a computer program read therefrom can be installed into the storage portion 908 as needed.

[0167] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the partition-based data migration method according to the embodiments of the present disclosure.

[0168] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0169] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to enable the computer system to implement the partition-based data migration method provided by the embodiments of the present disclosure.

[0170] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the processor 901 executes the computer program. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0171] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0172] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from a removable medium 911. When the computer program is executed by the processor 901, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0173] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0174] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0175] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.

[0176] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A partition-based data migration method, applied to a multi-cluster distributed system, characterized in that: The data migration method includes: Determine the data to be migrated and the target migration partition based on the data migration configuration file; Performing validity verification on the target migration partition; When it is determined that the target migration partition is valid, obtaining a data migration direction, where the data migration direction represents a data migration direction between multiple clusters; Executing a migration data acquisition script according to the data migration direction to acquire data to be migrated; and A data migration transmission script is executed according to the data to be migrated, the target migration partition, and the data migration direction to perform data migration at a partition granularity.

2. The data migration method according to claim 1, characterized in that: Executing the migration data acquisition script according to the data migration direction to obtain the data to be migrated includes: Generate a migration data acquisition script according to the data information to be migrated; Verify the cluster where the partition to be migrated is located according to the data migration direction; Save the data to be migrated locally according to the partition granularity.

3. The data migration method according to claim 2, wherein: Generating a migration data acquisition script according to the to-be-migrated data information includes: Traversing the data information to be migrated to determine a target input partition; generating a first target file according to the target input partition; and Generate a migration data acquisition script according to the partition range recorded in the first target file.

4. The data migration method according to claim 3, wherein: The data information to be migrated includes the name of the database to be migrated, the name of the data table to be migrated, and the name of the partition to be migrated. Traversing the data information to be migrated to determine the target input partition includes: Traverse the partition names to be migrated to determine the available range of partitions to be migrated; A target input partition is determined according to the available range of partitions to be migrated.

5. The data migration method according to claim 1, wherein: The executing the data migration transmission script according to the data to be migrated, the target migration partition, and the data migration direction to perform partition-granular data migration includes: generating a data migration transfer script according to the data to be migrated and the target migration partition; The data migration transmission script is executed according to the data migration direction to perform data migration at the partition granularity.

6. The data migration method according to claim 5, characterized in that: Generating a data migration transfer script according to the data to be migrated and the target migration partition includes: Traversing the acquired data directory according to the data to be migrated to generate a second target file; Generate a data migration and transmission script based on the data of the second target file.

7. The data migration method according to claim 6, characterized in that: Before executing the migration data transfer script, the following steps are also included: Generate a corresponding hive partition script according to the second target file; Execute the hive partition script to create hive partitions.

8. The data migration method according to any one of claims 1 to 7, characterized in that: The performing validity check on the target migration partition includes: Determine a target cluster according to the target migration partition; When it is determined that the target migration partition exists in the target cluster, determining that the target migration partition is invalid; and When it is determined that the target migration partition does not exist in the target cluster, the target migration partition is determined to be valid.

9. A partition-based data migration device, characterized in that: The device comprises: A first determining module is used to determine the data information to be migrated and the target migration partition according to the data migration configuration file; A verification module, configured to verify the validity of the target migration partition; A first acquisition module is configured to acquire a data migration direction when it is determined that the target migration partition is valid, where the data migration direction represents a data migration direction between multiple clusters; A second acquisition module is used to execute a migration data acquisition script according to the data migration direction to obtain the data to be migrated; The transmission module is configured to execute a data migration transmission script according to the data to be migrated, the target migration partition, and the data migration direction, so as to perform data migration at a partition granularity.

10. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the data migration method according to any one of claims 1 to 8.

11. A computer-readable storage medium having executable instructions stored thereon, wherein when the instructions are executed by a processor, the processor is caused to execute the data migration method according to any one of claims 1 to 8. 12 . A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the data migration method according to claim 1 is implemented.

Images