Big data migration method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a big data migration method and device, computer equipment, a readable storage medium and a program product. The method comprises the steps of determining target metadata corresponding to a first big data cluster based on a data migration instruction, generating a target structured query statement according to the target metadata, reading target data corresponding to the target metadata from the first big data cluster according to the target structured query statement through a distributed structured query language gateway, and sending the target data to the first big data cluster. Migrating the target data to the second big data cluster, and migrating the target metadata to the second distributed metadatabase. Compared with a traditional migration mode through a distributed replication tool, according to the scheme, through the distributed structured query language gateway, the target structured query statement generated based on the target metadata is migrated to the new cluster during data migration, and meanwhile, the metadata is migrated to the distributed metadatabase of the new big data cluster; and the big data migration efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a big data migration method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the development of communication technology, the amount of data generated in the communication field is huge, and the management and storage of communication data need to be performed through a distributed big data cluster. When managing communication big data, the historical data of an old cluster needs to be migrated to a new big data cluster. At present, the migration of communication big data from the old cluster to the new cluster is usually performed through a distributed replication tool. However, the migration of big data through the distributed replication tool needs to separately migrate metadata, which reduces the migration efficiency.

[0003] Therefore, the current migration method of communication big data has the defect of low migration efficiency. SUMMARY

[0004] Therefore, it is necessary to provide a big data migration method, device, computer equipment, computer readable storage medium and computer program product capable of improving migration efficiency in view of the above technical problems.

[0005] In a first aspect, the present application provides a big data migration method, comprising:

[0006] In response to a data migration instruction, target metadata corresponding to a first big data cluster is acquired.

[0007] According to the target metadata, a target structured query language (SQL) statement is generated.

[0008] The target SQL statement is sent to a distributed SQL gateway.

[0009] The distributed SQL gateway is configured to read target data corresponding to the target metadata from the first big data cluster according to the target SQL statement, migrate the target data to a second big data cluster, and migrate the target metadata to a second distributed metadata database corresponding to the second big data cluster.

[0010] In a second aspect, the present application further provides a big data migration device, comprising:

[0011] An acquisition module is configured to acquire target metadata corresponding to a first big data cluster in response to a data migration instruction.

[0012] A generation module is configured to generate a target structured query language (SQL) statement according to the target metadata.

[0013] a migration module configured to send the target structured query statement to a distributed structured query language gateway;

[0014] The distributed structured query language gateway is configured to read target data corresponding to the target metadata from the first big data cluster according to the target structured query statement, migrate the target data to a second big data cluster, and migrate the target metadata to a second distributed metadata database corresponding to the second big data cluster.

[0015] In a third aspect, the present application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described above when executing the computer program.

[0016] In a fourth aspect, the present application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method described above.

[0017] In a fifth aspect, the present application also provides a computer program product including a computer program, wherein the computer program is executed by a processor to implement the steps of the method described above.

[0018] The big data migration method, device, computer device, computer readable storage medium and computer program product described above determine target metadata corresponding to a first big data cluster based on a data migration instruction, generate a target structured query statement according to the target metadata, read target data corresponding to the target metadata from the first big data cluster according to the target structured query statement through a distributed structured query language gateway, migrate the target data to a second big data cluster, and migrate the target metadata to a second distributed metadata database. Compared with the traditional migration mode through a distributed replication tool, the present application migrates to a new cluster through a target structured query statement generated based on target metadata through a distributed structured query language gateway during data migration, and simultaneously migrates metadata to a distributed metadata database of a new big data cluster, thereby improving the efficiency of big data migration. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0020] Figure 1 a flowchart of the big data migration method in an embodiment;

[0021] Figure 2 a flowchart of a large data migration method in another embodiment;

[0022] Figure 3 a structural block diagram of a large data migration device in an embodiment;

[0023] Figure 4 an internal structural diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0024] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0025] It should be noted that the terms "first", "second", etc. used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" used in the present application and any variations thereof are intended to cover non-exclusive inclusion. The term "a plurality of" used in the present application means two and more than two. The term "and / or" used in the present application means one of the schemes or any combination of a plurality of schemes.

[0026] In the related art, after the construction of a new communication big data cluster, it is necessary to quickly and accurately migrate the historical data of the old big data cluster to the newly built big data cluster. For example, to the big data cluster of a distributed computing framework. Among them, the number of files of the communication big data is large, which puts a lot of pressure on the component responsible for managing the namespace of the distributed file system. If the traditional data copying tool (such as the discp command) is used for data copying, not only is the efficiency low, but also the problem of small files in the old cluster is passed to the new cluster, which brings cumbersome work to the subsequent cluster governance, and the migration of data through the distributed data replication tool also needs to separately migrate the metadata, and then perform additional work such as partition data repair.

[0027] Based on this, the present application generates a target structured query statement based on target metadata through a distributed structured query language gateway during data migration, migrates to a new cluster, and migrates metadata to a distributed metadata database of a new big data cluster. The enterprise-level data lake management platform is used for cross-cluster migration, and the data file and metadata are migrated at one time. When writing to the new cluster, small files are merged according to the target file size. The enterprise-level data lake management platform uses the spark-sql engine for data migration, which can improve the migration efficiency, and according to the target file size, the data file and metadata are migrated to the new cluster at one time, reducing the additional work and improving the efficiency of big data migration.

[0028] In one embodiment, as shown in Figure 1 A big data migration method is provided, and the present embodiment is exemplified by the method applied to a server. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and can be realized through the interaction of the terminal and the server, including the following steps S202 to S206. Among them:

[0029] Step S202, in response to a data migration instruction, obtaining target metadata corresponding to a first big data cluster.

[0030] The server can be a server for big data migration. The server can be deployed with a data migration service for big data migration. The data migration instruction can be an instruction for indicating the data migration of the big data cluster. For example, migrating the data of the first big data cluster to the second big data cluster. The first big data cluster can be an old big data cluster, and the second big data cluster can be a newly built big data cluster. The data migration instruction can be triggered by the administrator of the data cluster, and the server can obtain the target metadata corresponding to the first big data cluster in response to the data migration instruction. The target metadata represents the metadata corresponding to the data to be migrated.

[0031] The first big data cluster can include multiple partitions and multiple tables. The table can be a data table storing data. Partitioning means physically dividing a large table into multiple smaller parts, which are called partitions. Although logically, the partitioned table still looks like a single table, at the physical level, each partition is stored on an independent file, which can be located on different hard disks of the same server or distributed across multiple servers. Not all data in the first big data cluster is migrated, and the server can determine the partitions and tables that need to be migrated in the first big data cluster based on the data migration instruction, and obtain the target metadata corresponding to the data in these partitions and tables that need to be migrated.

[0032] The server can obtain the target metadata from the first distributed metadata database. The first distributed metadata database can be a database corresponding to the first big data cluster, and the first distributed metadata database stores metadata corresponding to data in the first big data cluster. Specifically, the first distributed metadata database can be a Metastore, and the first big data cluster can be connected to the Metastore corresponding to the first big data cluster. The Metastore is used to store table structures, column information, partition information, data storage locations, and other metadata.

[0033] In step S204, the target structured query statement is generated according to the target metadata.

[0034] After the server obtains the target metadata, it determines which data needs to be migrated. The server can generate a target structured query statement based on the target metadata. The target structured query statement represents a statement for migrating target data corresponding to the target metadata from the old cluster to the new cluster. The server can use the target structured query statement to query the target data that needs to be migrated from the first big data cluster, and migrate the queried target data to the new cluster. The server can cache the target data that needs to be migrated based on the target metadata, such as caching the partitions and tables where the target data is stored, and based on the cached partitions and tables, the target data is migrated in combination with the target structured query statement.

[0035] The target structured query statement can be generated based on whether the data table is a partitioned table. That is, for partitioned tables and non-partitioned tables, the server can generate different target structured query statements. A partitioned table means that the data of the data table is divided into multiple partitions for storage, and a non-partitioned table means that the data of the data table is stored in one partition.

[0036] In step S206, the target structured query statement is sent to a distributed structured query language gateway. The distributed structured query language gateway is used to read the target data corresponding to the target metadata from the first big data cluster based on the target structured query statement, migrate the target data to the second big data cluster, and migrate the target metadata to the second distributed metadata database corresponding to the second big data cluster.

[0037] The server can be connected to the distributed structured query language gateway, and the distributed structured query language gateway can be connected to the first big data cluster, the first distributed metadata database, the second big data cluster, and the second distributed metadata database, respectively. The second distributed metadata database can be a database corresponding to the second big data cluster for storing metadata.

[0038] After the server obtains the target structured query statement, the server can send the target structured query statement to the distributed query language gateway. Thus, the distributed structured query language gateway can perform data query in the first big data cluster using the target structured query statement, read target data corresponding to the target metadata, migrate the target data to the second big data cluster, and migrate the target metadata to the second distributed metadata database corresponding to the second big data cluster, thereby realizing migration of big data.

[0039] The distributed structured query language gateway can be Kyuubi. Kyuubi is a distributed multi-tenant structured query language (SQL) gateway that provides efficient SQL query services for big data processing and aims to solve complex problems of data access and processing through SQL in a big data scenario. The distributed structured query language gateway can be configured with a spark application. The distributed structured query language gateway can start the spark application on the cluster to implement data computing tasks and thus realize migration of target data. The spark is a distributed parallel computing framework based on memory.

[0040] In the big data migration method, the target metadata corresponding to the first big data cluster is determined based on the data migration instruction, the target structured query statement is generated based on the target metadata, the target data corresponding to the target metadata is read from the first big data cluster based on the target structured query statement, the target data is migrated to the second big data cluster, and the target metadata is migrated to the second distributed metadata database. Compared with the traditional migration method through a distributed replication tool, the present scheme migrates to a new cluster through a target structured query statement generated based on target metadata and migrates metadata to a distributed metadata database of a new big data cluster during data migration through a distributed structured query language gateway, thereby improving the efficiency of big data migration.

[0041] In one embodiment, the target structured query statement is generated based on the target metadata. If a data table corresponding to the target metadata does not exist in the second big data cluster, a data table corresponding to the target metadata is generated in the second big data cluster based on the target metadata. The target structured query statement is generated based on the data table corresponding to the target metadata.

[0042] In this embodiment, the server can detect whether the data table corresponding to the target metadata exists in the second big data cluster, i.e., the new big data cluster, when migrating the target data. If the server detects that the data table corresponding to the target metadata does not exist in the second big data cluster, the server can generate the data table corresponding to the target metadata in the second big data cluster according to the target metadata. The generated data table can be the same as the data table corresponding to the target metadata in the first big data cluster.

[0043] Therefore, after determining that the data table corresponding to the target metadata exists in the second big data cluster, the server can generate the target structured query statement according to the data table corresponding to the target metadata. For example, the terminal can use the table name of the target data table as the table name of the new table into which the target data is inserted in the target structured query statement, so as to migrate the target data from the first big data cluster to the target data table in the second big data cluster through the target structured query statement.

[0044] Specifically, the server uses Java Database Connectivity (JDBC) to connect the metastore database to obtain which tables and partitions need to be migrated, and caches the partitions and tables that need to be migrated in a collection. The server migrates the cached table information through a data migration service. Each time a data table needs to be migrated to the second big data cluster, the server needs to first check whether the data table corresponding to the target metadata exists in the second big data cluster. If the data table exists, the server directly generates a target structured query statement in combination with the information of the data table corresponding to the target metadata. If the data table does not exist, the server creates a data table in the second big data cluster that is consistent with the data table corresponding to the target metadata in the first big data cluster, and performs data migration in combination with the newly created data table.

[0045] Through this embodiment, the server detects whether the data table corresponding to the target metadata exists in the second big data cluster, and creates a data table in the second big data cluster that is consistent with the table in the old cluster in a timely manner when the data table does not exist, thereby improving the efficiency of big data migration.

[0046] In one embodiment, generating the target structured query statement according to the target metadata includes: if the data table corresponding to the target metadata is not a partitioned table, generating a first target structured query statement according to a data table identifier in the target metadata; the partitioned table represents a data table in which data is stored in different physical locations; and if the data table corresponding to the target metadata is a partitioned table, generating a second target structured query statement according to the data table identifier and a partition identifier in the target metadata.

[0047] In this embodiment, for the data table that needs to be migrated, the server can detect whether the data table belongs to a partition table, which represents a data table whose data is stored in different physical locations. For the data table belonging to the partition table and the data table not belonging to the partition table, the server can generate different target structured query statements. For example, for the data table not belonging to the partition table, the server can generate a first target structured query statement according to the data table identifier in the target metadata, without considering the partition of different data in the data table during data migration. The data table identifier can include the table name of the data table. The server can take the data table identifier of the target metadata in the first big data cluster as the data table identifier of the migrated-out data, take the data table identifier of the target metadata in the second big data cluster as the data table identifier of the migrated-in data, and then generate the first target structured query statement.

[0048] For the data table belonging to the partition table, the server can generate a second target structured query statement according to the data table identifier and the partition identifier in the target metadata, considering the partition of different data in the data table during data migration. The partition identifier represents the identifier of the partition where the data in the data table is located, such as the name of the partition. The server can take the data table identifier and the partition identifier of the target metadata in the first big data cluster as the data table identifier and the partition identifier of the migrated-out data, take the data table identifier and the partition identifier of the target metadata in the second big data cluster as the data table identifier and the partition identifier of the migrated-in data, and then generate the second target structured query statement.

[0049] Specifically, the server can generate a structured query statement (target structured query statement) of spark for each migrated table or partition of a table, and use different statement generation methods for partitioned tables and non-partitioned tables. For example, for a non-partitioned table, the first target structured query statement can be represented as: insert into table db.tbl select * from old_catalog.db.tbl. Wherein, "db.tbl" represents the identifier of the data table corresponding to the target metadata. "insert into table db.tbl" represents appending data to the data table corresponding to the target metadata in the second big data cluster; "select * from old_catalog.db.tbl" represents querying all fields and all data from the same table (db.tbl) under the catalog (old_catalog) in the first big data cluster. Wherein, "select *" represents all fields, and the number of fields, the field type, and the field order of the data table in the first big data cluster and the data table in the second big data cluster are completely consistent.

[0050] For the partition table, the second target structured query statement can be expressed as: insert into table db.tblpartition(partition_name="partition value") select * from old_catalog.db.tbl where partition_name="partition value". Wherein, the partition table represents that the data table "db.tbl" is stored according to the specified field partition_name (such as date, region), different partitions correspond to different physical directories, and the partition field can be filtered during query. "Insert into table db.tbl partition(partition_name="partition value")" indicates that data is added to the specified partition of the target metadata corresponding partition table in the second big data cluster, and the partition value identifies the specific field of the partition, indicating which partition the data needs to be written to. "Select * from old_catalog.db.tbl where partition_name="partition value"" indicates that only the target data of the partition corresponding to "partition value" is queried from the data table corresponding to the target metadata in the first big data cluster.

[0051] Through the embodiment, the server can detect whether the data table corresponding to the target metadata belongs to a partition table, and generate target structured query statements corresponding to the partition table and the non-partition table respectively in different ways, so as to perform data migration on different types of data tables by using different statements, thereby improving the efficiency of big data migration.

[0052] In one embodiment, the above target data is migrated to the second big data cluster, including: if there is target data with a data volume less than a data volume threshold, merging each target data less than the data volume threshold to obtain merged target data greater than or equal to the data volume threshold; and migrating the merged target data to the target data table corresponding to the target metadata in the second big data cluster.

[0053] In the embodiment, when the server performs big data cluster migration, there are many small files in the first big data cluster. In order to avoid file management problems after data migration, the server can merge the scattered small files during data migration, for example, merge data belonging to the same type but stored in different servers in the first big data cluster, thereby reducing data redundancy.

[0054] The server can set a data amount threshold, which can be set according to actual conditions. When migrating the target data to the second big data cluster, if the server detects that there is target data with a data amount less than the data amount threshold, the server can merge each target data less than the data amount threshold to obtain merged target data greater than or equal to the data amount threshold. Thus, the server can migrate the merged target data to the target data table corresponding to the target metadata in the second big data cluster.

[0055] Specifically, the distributed structured query language gateway can configure a catalog for cross-metabase access, and configure the last state of the distributed structured query language gateway in the data migration process, and optimize the state of the data after data migration, for example, configure the target file size (data amount threshold), configure that the data will be redistributed between nodes and the distribution operation (such as shuffle operation) will be re-executed when data is inserted. Among them, redistribution refers to redistributing data to different nodes according to rules in distributed computing.

[0056] The server can start multiple threads for data table migration, and the number of threads started can be configured according to actual conditions. The server submits the generated target structured query statement to the distributed structured query language gateway for execution. The distributed structured query language gateway starts the spark application on the cluster when migrating, and when writing the target data in the first big data cluster to the corresponding data table in the second big data cluster, the small files will be merged and stored according to the data amount threshold configured above.

[0057] Through the embodiment, the server can merge small files according to the set data amount threshold when migrating big data, thereby reducing the difficulty of data management after data migration and improving the efficiency of data management.

[0058] In one embodiment, after sending the target structured query statement to the distributed structured query language gateway, the method further includes: obtaining a migration result returned by the distributed structured query language gateway; generating a migration record according to the target metadata and the migration result, and storing the migration record to the relational database management system.

[0059] In this embodiment, after sending the target structured query statement to the distributed structured query language gateway, the server can also continuously track and record the migration result of the data migration by the distributed structured query language gateway. For example, the server can obtain the migration result returned by the distributed structured query language gateway. The migration result includes whether the data migration by the distributed structured query language gateway is completed, what data fails to be migrated, the time used for the migration, and the like. Thus, the server can generate a migration record according to the target metadata and the migration result, and store the migration record to the relational database management system.

[0060] Specifically, for the data table or partition that succeeds in the migration or fails in the migration, the server can obtain the corresponding migration result and record the migration result to the relational database management system, facilitating subsequent verification of the migration record.

[0061] Through this embodiment, the server can record the migration result of the data migration of the big data cluster, thereby improving the efficiency of subsequent migration record management.

[0062] In one embodiment, in response to the data migration instruction, the method further includes: sending an authentication instruction to the first big data cluster and the second big data cluster respectively; the authentication instruction is used to instruct the first big data cluster and the second big data cluster to perform authentication based on a key distribution protocol; if the first big data cluster and the second big data cluster pass the authentication, the target metadata corresponding to the first big data cluster is obtained.

[0063] In this embodiment, when performing the data migration of the big data cluster, the server can pre-authenticate the first big data cluster and the second big data cluster with each other. After passing the mutual authentication, the first big data cluster and the second big data cluster can realize data communication. For example, the server can send an authentication instruction to the first big data cluster and the second big data cluster respectively. The first big data cluster can perform authentication with the second big data cluster based on a key distribution protocol according to the authentication instruction. If the first big data cluster and the second big data cluster pass the authentication, it indicates that the first big data cluster can migrate data to the second big data cluster, and thus the server can obtain the target metadata corresponding to the first big data cluster.

[0064] Specifically, the key distribution protocol described above can be a Kerberos protocol based on a key distribution center (KDC), and the Kerberos protocol can be a network authentication protocol. Between the old first big data cluster and the new second big data cluster, the key distribution center mutual authentication can be configured. The key distribution center is a service for managing and distributing encryption keys in the field of network security, especially in systems using the Kerberos protocol. After mutual authentication between the first big data cluster and the second big data cluster, big data migration can be performed.

[0065] Through the embodiment, the server can perform mutual authentication between the old and new big data clusters, and only allow big data migration after the mutual authentication is passed, thereby improving the security of data migration.

[0066] In one exemplary embodiment, as shown in Figure 2 , Figure 2 is a flowchart of the big data migration method in another embodiment.

[0067] In the embodiment, the key distribution center mutual authentication can be configured between the old first big data cluster and the new second big data cluster. The key distribution center is a service for managing and distributing encryption keys in the field of network security, especially in systems using the Kerberos protocol. After mutual authentication between the first big data cluster and the second big data cluster, big data migration can be performed.

[0068] The distributed structured query language gateway can configure a catalog for cross-metabase access, and configure the last state of the distributed structured query language gateway in the data migration process, and optimize the state of the data after data migration, for example, configure the target file size (data volume threshold), configure that the data will not be redistributed between nodes when inserted, and the distribution operation (such as shuffle operation) will also be executed again. Redistribute refers to redistributing data to different nodes according to rules in distributed computing.

[0069] The data migration service is configured in the server. The server uses the Java Database Connectivity (JDBC) to connect the metastore database through the data migration service, obtains which tables and partitions need to be migrated, and caches the partitions and tables that need to be migrated in a collection. The server migrates the table information just cached through the data migration service. Each time a data table needs to be migrated to the second big data cluster, the server needs to see whether the target metadata corresponding data table exists in the second big data cluster. If it exists, the server directly generates a target structured query statement in combination with the information of the target metadata corresponding data table. If it does not exist, the server creates a data table consistent with the target metadata corresponding data table in the first big data cluster in the second big data cluster, and performs data migration in combination with the newly created data table.

[0070] The server can generate a spark structured query statement (target structured query statement) for each table or partition of a table migrated, and adopt different statement generation modes for partitioned tables. For example, for a non-partitioned table, the first target structured query statement can be expressed as: insert into table db.tbl select * from old_catalog.db.tbl. Wherein, "db.tbl" represents the identifier of the data table corresponding to the target metadata. "insert into table db.tbl" represents appending data to the data table corresponding to the target metadata in the second big data cluster; "select * from old_catalog.db.tbl" represents querying all fields and all data from the same table (db.tbl) under the catalog (old_catalog) in the first big data cluster. Wherein, "select *" represents all fields, and the number of fields, field types, and field orders of the data table in the first big data cluster and the data table in the second big data cluster are completely consistent.

[0071] For the partition table, the second target structured query statement can be expressed as: insert into table db.tblpartition(partition_name="partition value") select * from old_catalog.db.tbl where partition_name="partition value". Wherein, the partition table represents that the data table "db.tbl" is stored according to the specified field partition_name (such as date, region), different partitions correspond to different physical directories, and the partition field can be filtered during query. "Insert into table db.tbl partition(partition_name="partition value")" indicates that data is added to the specified partition of the partition table corresponding to the target metadata in the second big data cluster, the partition value identifies the specific field of the partition, and indicates which partition needs to write data. "Select * from old_catalog.db.tbl where partition_name="partition value"" indicates that only the target data of the partition corresponding to "partition value" is queried from the data table corresponding to the target metadata in the first big data cluster.

[0072] The server can start multiple threads for data table migration, and the number of started threads can be configured according to actual conditions. The server submits the generated target structured query statement to the distributed structured query language gateway for execution. The distributed structured query language gateway starts the spark application on the cluster during migration, and at the same time, when writing the target data in the first big data cluster into the corresponding data table in the second big data cluster, the small files are merged and stored according to the data volume threshold configured above. Wherein, for the data table or partition that is successfully migrated or failed to migrate, the server can obtain the corresponding migration result and record it to the relational database management system, facilitating subsequent verification of the migration record.

[0073] Through the embodiment, the server migrates to the new cluster through the target structured query statement generated based on the target metadata through the distributed structured query language gateway during data migration, and at the same time, the metadata is migrated to the distributed metadata database of the new big data cluster, thereby improving the efficiency of big data migration.

[0074] It should be understood that although each step in the flowchart involved in the embodiments described above is shown in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowchart involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.

[0075] Based on the same inventive concept, the embodiments of the present application also provide a big data migration device for implementing the above-mentioned big data migration method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more big data migration device embodiments provided below can refer to the limitations of the big data migration method described above, which will not be repeated here.

[0076] In one exemplary embodiment, as shown in Figure 3 A big data migration device is provided, comprising: an acquisition module 500, a generation module 502, and a migration module 504, wherein:

[0077] The acquisition module 500 is configured to acquire target metadata corresponding to a first big data cluster in response to a data migration instruction.

[0078] The generation module 502 is configured to generate a target structured query statement according to the target metadata.

[0079] The migration module 504 is configured to send the target structured query statement to a distributed structured query language gateway. The distributed structured query language gateway is configured to read target data corresponding to the target metadata from the first big data cluster according to the target structured query statement, migrate the target data to a second big data cluster, and migrate the target metadata to a second distributed metadata database corresponding to the second big data cluster.

[0080] In one embodiment, the generation module 502 is configured to, if the data table corresponding to the target metadata does not exist in the second big data cluster, generate the data table corresponding to the target metadata in the second big data cluster according to the target metadata; and generate the target structured query statement according to the data table corresponding to the target metadata.

[0081] In one embodiment, the generation module 502 is configured to, if the data table corresponding to the target metadata does not belong to a partition table, generate a first target structured query statement according to the data table identifier in the target metadata; and if the data table corresponding to the target metadata belongs to a partition table, generate a second target structured query statement according to the data table identifier and the partition identifier in the target metadata, wherein the partition table represents a data table in which data is stored in different physical locations.

[0082] In one embodiment, the migration module 504 is configured to, if there is target data with a data amount less than a data amount threshold, merge each target data less than the data amount threshold to obtain merged target data greater than or equal to the data amount threshold; and migrate the merged target data to the target data table corresponding to the target metadata in the second big data cluster.

[0083] In one embodiment, the apparatus further includes a recording module configured to acquire the migration result returned by the distributed structured query language gateway; generate a migration record according to the target metadata and the migration result; and store the migration record to a relational database management system.

[0084] In one embodiment, the apparatus further includes an authentication module configured to send an authentication instruction to the first big data cluster and the second big data cluster, respectively; the authentication instruction is used to instruct the first big data cluster and the second big data cluster to perform authentication based on a key distribution protocol; and if the first big data cluster and the second big data cluster pass the authentication, acquire the target metadata corresponding to the first big data cluster.

[0085] The modules in the big data migration apparatus can be all or partially implemented by software, hardware, and combinations thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to the modules.

[0086] In one exemplary embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in FIG. 1. Figure 4As shown in the figure. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store communication data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a big data migration method.

[0087] Those skilled in the art can understand that, Figure 4 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0088] In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-mentioned big data migration method.

[0089] In one embodiment, a computer readable storage medium is provided, having a computer program stored thereon, and the computer program is executed by the processor to implement the above-mentioned big data migration method.

[0090] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by the processor to implement the above-mentioned big data migration method.

[0091] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0092] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0093] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0094] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A method for migrating large data, the method comprising: The method comprises: in response to a data migration instruction, obtaining target metadata corresponding to a first big data cluster; generating a target structured query language statement according to the target metadata; sending the target structured query language statement to a distributed structured query language gateway; the distributed structured query language gateway is configured to read target data corresponding to the target metadata from the first big data cluster according to the target structured query language statement, migrate the target data to a second big data cluster, and migrate the target metadata to a second distributed metadata database corresponding to the second big data cluster.

2. The method of claim 1, wherein, The generating of the target structured query language statement according to the target metadata comprises: if a data table corresponding to the target metadata does not exist in the second big data cluster, generating the data table corresponding to the target metadata in the second big data cluster according to the target metadata; generating a target structured query language statement according to the data table corresponding to the target metadata.

3. The method of claim 1, wherein, The generating of the target structured query language statement according to the target metadata comprises: if the data table corresponding to the target metadata does not belong to a partition table, generating a first target structured query language statement according to a data table identifier in the target metadata; the partition table represents a data table in which data is stored in different physical locations; if the data table corresponding to the target metadata belongs to a partition table, generating a second target structured query language statement according to the data table identifier and a partition identifier in the target metadata.

4. The method of claim 1, wherein, The migrating of the target data to the second big data cluster comprises: if there is target data with a data volume less than a data volume threshold, merging each target data less than the data volume threshold to obtain merged target data greater than or equal to the data volume threshold; migrating the merged target data to a target data table corresponding to the target metadata in the second big data cluster.

5. The method of claim 1, wherein, After the sending of the target structured query language statement to the distributed structured query language gateway, the method further comprises: obtaining a migration result returned by the distributed structured query language gateway; generating a migration record according to the target metadata and the migration result, and storing the migration record to a relational database management system.

6. The method according to any one of claims 1 to 5, characterized in that, After the response to the data migration instruction, the method further comprises: sending an authentication instruction to the first big data cluster and the second big data cluster respectively; the authentication instruction is configured to instruct the first big data cluster and the second big data cluster to perform authentication based on a key distribution protocol; if the first big data cluster and the second big data cluster pass the authentication, obtaining target metadata corresponding to the first big data cluster.

7. A big data migration apparatus, characterized by, The apparatus comprises: an obtaining module configured to obtain target metadata corresponding to a first big data cluster in response to a data migration instruction; a generating module configured to generate a target structured query language statement according to the target metadata; a migrating module configured to send the target structured query language statement to a distributed structured query language gateway; The distributed structured query language gateway is configured to read target data corresponding to the target metadata from the first big data cluster according to the target structured query statement, migrate the target data to a second big data cluster, and migrate the target metadata to a second distributed metadata database corresponding to the second big data cluster.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6. The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

Images