A method and device for data migration

Abstract

The present invention discloses a method and device for data migration, and relates to the field of computer technology. A specific implementation of the method includes: obtaining a data table to be migrated from a source database according to a data source name; extracting target data from the data table to be migrated based on table fields; mapping the table fields to target fields, and converting the type of the target data to the target type to generate migration data; and storing the migration data in the target database. This implementation can efficiently migrate massive data; and in the process of migrating massive data, the massive data is secondary processed to be processed into data that meets the needs of analysis or search.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for data migration. Background Art

[0002] At present, the development of most enterprises is driven by data analysis, using big data analysis to achieve insights and fully unleash the potential of enterprises. However, the storage, analysis and migration of large amounts of data have always been a challenge. With the explosive growth of data volume, the time for rapid analysis and processing of big data has become increasingly stringent. The existing data migration method is to use the most primitive means to store the data exported from Hive in .csv format, and then import the file data into ClickHouse through the command line of ClickHouse Client.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] The migration efficiency is low and requires a lot of effort; and the migrated data also needs to be processed again to meet the needs. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and device for data migration, which can efficiently migrate massive data; and in the process of migrating massive data, the massive data is secondary processed to process it into data that meets the needs of analysis or search.

[0006] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for data migration is provided.

[0007] A data migration method according to an embodiment of the present invention includes:

[0008] Get the data table to be migrated from the source database according to the data source name;

[0009] Extracting target data from the to-be-migrated data table based on table fields;

[0010] Mapping the table fields to target fields and converting the type of the target data to the target type to generate migration data;

[0011] The migration data is stored in the target database.

[0012] Optionally, extracting target data from the to-be-migrated data table based on table fields includes:

[0013] Extracting intermediate data from the to-be-migrated data table based on table fields;

[0014] The target data is filtered out from the intermediate data using the value of the table field.

[0015] Optionally, storing the migration data in a target database includes:

[0016] Based on the data source name, query whether there is a storage data table storing the migration data in the target database; wherein the storage data table corresponds to the data source name;

[0017] If it does not exist, create the storage data table based on the target field; and insert the migration data into the storage data table of the sub-cluster of the target database; wherein the storage data table is distributed and stored in each sub-cluster of the target database;

[0018] If so, insert the migration data into the storage data table of the sub-cluster of the target database.

[0019] Optionally, inserting the migration data into the storage data table of the sub-cluster of the target database includes:

[0020] Performing a modulo operation on the hash value of the data source name and the date of the migration data to obtain a modulo remainder;

[0021] matching a target subcluster from the subcluster of the target database based on the modulus;

[0022] Insert the migration data into the target subcluster.

[0023] Optionally, the source database is a Hive data warehouse, the target database is ClickHouse, and

[0024] Get the data table to be migrated from the source database according to the data source name, including:

[0025] Using Spark, a data table to be migrated is obtained from a Hive data warehouse according to at least one database name and / or data table name; wherein the data table to be migrated is a matrix data table having a row index and a column index.

[0026] To achieve the above objective, according to another aspect of an embodiment of the present invention, a device for migrating data is provided.

[0027] A data migration device according to an embodiment of the present invention includes:

[0028] The acquisition module is used to obtain the data table to be migrated from the source database according to the data source name;

[0029] An extraction module, used for extracting target data from the to-be-migrated data table based on table fields;

[0030] A generation module, used for mapping the table fields to target fields and converting the type of the target data to the target type to generate migration data;

[0031] The storage module is used to store the migration data in a target database.

[0032] Optionally, the extraction module is further used for:

[0033] Extracting intermediate data from the to-be-migrated data table based on table fields;

[0034] The target data is filtered out from the intermediate data using the value of the table field.

[0035] Optionally, the storage module is further used for:

[0036] Based on the data source name, query whether there is a storage data table storing the migration data in the target database; wherein the storage data table corresponds to the data source name;

[0037] If it does not exist, create the storage data table based on the target field; and insert the migration data into the storage data table of the sub-cluster of the target database; wherein the storage data table is distributed and stored in each sub-cluster of the target database;

[0038] If so, insert the migration data into the storage data table of the sub-cluster of the target database.

[0039] Optionally, the storage module is further used for:

[0040] Performing a modulo operation on the hash value of the data source name and the date of the migration data to obtain a modulo remainder;

[0041] matching a target subcluster from the subcluster of the target database based on the modulus;

[0042] Insert the migration data into the target subcluster.

[0043] Optionally, the source database is a Hive data warehouse, the target database is ClickHouse, and

[0044] The acquisition module is also used for:

[0045] Using Spark, a data table to be migrated is obtained from a Hive data warehouse according to at least one database name and / or data table name; wherein the data table to be migrated is a matrix data table having a row index and a column index.

[0046] To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device for data migration is provided.

[0047] An electronic device for data migration according to an embodiment of the present invention comprises: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement a method for data migration according to an embodiment of the present invention.

[0048] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided.

[0049] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, and when the program is executed by a processor, a data migration method according to an embodiment of the present invention is implemented.

[0050] An embodiment of the above invention has the following advantages or beneficial effects: because it adopts the technical means of obtaining the data table to be migrated from the source database according to the data source name; extracting the target data from the data table to be migrated based on the table fields; mapping the table fields to the target fields, and converting the type of the target data to the target type to generate the migration data; storing the migrated data in the target database, it overcomes the technical problems of low migration efficiency and requiring a lot of effort; and the migrated data also needing secondary processing to meet the needs, thereby achieving the ability to efficiently migrate massive data; and in the process of migrating massive data, the massive data is subjected to secondary processing and processed into data that meets the needs of analysis or search, etc.

[0051] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0053] Figure 1 is a schematic diagram of main steps of a method for data migration according to an embodiment of the present invention;

[0054] Figure 2 is a schematic diagram of an implementation process of a method for data migration according to a reference embodiment of the present invention;

[0055] Figure 3 is a schematic diagram of an implementation framework of a method for data migration according to an embodiment of the present invention;

[0056] Figure 4 is an application diagram of the method for data migration according to an embodiment of the present invention. Figure 1 ;

[0057] Figure 5 is an application diagram of the method for data migration according to an embodiment of the present invention. Figure 2 ;

[0058] Figure 6 is a schematic diagram of main modules of a method for data migration according to an embodiment of the present invention;

[0059] Figure 7 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0060] Figure 8 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0062] It should be pointed out that the embodiments of the present invention and the technical features therein may be combined with each other without conflict.

[0063] At present, for the situation of massive data, the efficiency of data query and analysis based on Hadoop ecosystem can no longer meet the business needs. Hadoop is a distributed system infrastructure that implements a distributed file system (HDFS). Users can develop distributed programs without understanding the underlying details of the distribution and make full use of the power of the cluster for high-speed computing and storage.

[0064] However, an efficient solution is required for data migration between different storage media. To this end, an embodiment of the present invention provides a method for data migration, which can perform secondary processing on massive data during the process of mass data migration to process the massive data into data that meets the needs of analysis or search.

[0065] Figure 1 is a schematic diagram of main steps of a method for data migration according to an embodiment of the present invention.

[0066] like Figure 1 As shown, the data migration method of the embodiment of the present invention mainly includes the following steps:

[0067] Step S101: obtaining a data table to be migrated from a source database according to a data source name.

[0068] At present, large amounts of data are mostly stored in data warehouses. When migrating such data, the location of the data to be migrated can be determined based on the data source name, and the data source name can be used to preliminarily screen the data in the source database to obtain the data tables to be migrated.

[0069] As a preferred implementation, the source database is a Hive data warehouse and the target database is ClickHouse.

[0070] Hive is a data warehouse tool based on Hadoop, which can map structured data files into a database table and provide SQL-like query functions. Hive data warehouse is a data warehouse based on Hive for query analysis. Clickhouse is a columnar database management system (columnar DBMS) for online analytical processing (OLAP), and its data tables are distributed in various subclusters. ClickHouse can meet the business needs for efficient data insertion and reading analysis.

[0071] In an embodiment of the present invention, step S101 may be implemented in the following manner: using Spark to obtain a data table to be migrated from a Hive data warehouse according to at least one database name and / or data table name.

[0072] The data table to be migrated is a matrix data table, which has a row index and a column index. The data therein can be filtered using the row index or the column index.

[0073] Spark is a memory-based big data distributed computing framework, a fast and general computing engine designed for large-scale data processing. Using Spark's parser (Spark SQL), you can read the data that needs to be migrated in the Hive data warehouse through the database connection interface (JDBCAPI) to form a matrix data table (DateFrame).

[0074] Step S102: extracting target data from the data table to be migrated based on the table fields.

[0075] The data in the data table to be migrated can be filtered using the row index or column index, that is, the row index or column index of the data table to be migrated can be filtered using the table field, and the table field can be a field representing the attributes, dimensions or characteristics of the target data.

[0076] In the embodiment of the present invention, step S102 may be implemented in the following manner: extracting intermediate data from the to-be-migrated data table based on table fields; and filtering out target data from the intermediate data using values ​​of the table fields.

[0077] In order to achieve secondary processing of data during the data migration process and process it into data that meets the needs of analysis or search, intermediate data with table fields can be extracted from the data table to be migrated, and then target data with specific values ​​(i.e., the values ​​of the table fields) can be screened out from the intermediate data. The target data is the data that ultimately needs to be migrated.

[0078] In addition, step S101 and step S102 can be automatically implemented through the configuration of Python scripts. Python is an object-oriented dynamic programming language that is often used to write automated scripts. Standard configuration scripts can control the directional migration of data to any ClickHouse. The data has extremely high efficiency during the migration process, ensuring the timeliness requirements of the migrated data. At the same time, the data can be processed again during the migration process to better serve the business.

[0079] Step S103: Map the table fields to the target fields, and convert the type of the target data to the target type to generate migration data.

[0080] Since the table fields of the same attribute, dimension or feature may be expressed completely differently in the source database and the target database, for example, the field "70KG" in the same data belongs to body weight and weight in the two databases respectively, so the table field can be mapped with the corresponding target field during migration. And the target data screened out by step S102 is data from the source database, and its data type may not meet the storage requirements or computing requirements of the target database, so the type of the target data can be converted during migration. The type of the target data can be byte, short integer, integer, long integer, single-precision floating point, double-precision floating point, etc., and the target type can also be byte, short integer, integer, long integer, single-precision floating point, double-precision floating point, etc.

[0081] Step S104: storing the migrated data in the target database.

[0082] After obtaining the migrated data, it can be stored in the target database for analysis, query, calculation, etc.

[0083] In an embodiment of the present invention, step S104 can be implemented in the following manner: query whether there is a storage data table storing migration data in the target database based on the data source name; if not, create a storage data table based on the target field; and insert the migration data into the storage data table of the subcluster of the target database; if it exists, insert the migration data into the storage data table of the subcluster of the target database.

[0084] The storage data table corresponds to the data source name, that is, the migration data from the same source database will be stored in the same storage data table. In the target database, the migration data is distributed and stored in each sub-cluster of the target database in the form of storage data tables. After obtaining the migration data through the above steps, check whether the storage data table exists in the target database. If it does not exist, create the table first and then insert the migration data. If it exists, insert the migration data directly.

[0085] In an embodiment of the present invention, the step of inserting the migration data into the storage data table of the subcluster of the target database can be implemented in the following manner: performing a modulus operation on the hash value of the data source name and the date of the migration data to obtain a modulus remainder; matching a target subcluster from the subcluster of the target database based on the modulus remainder; and inserting the migration data into the target subcluster.

[0086] To ensure the speed of data query, when storing migration data, the migration data can be evenly distributed on each sub-cluster of the target database by date, for example, the migration data of the same day is evenly distributed on each sub-cluster. Specifically, for each migration data, a modulo operation is performed using the hash value of the data source name and the date of the migration data, and the corresponding target sub-cluster is matched from all sub-clusters based on the obtained modulo remainder, and the migration data is inserted into the corresponding target sub-cluster. Modulo operation refers to modular division (also known as modulus, modulo operation, modulo operation, etc.), and the result is the remainder of one number divided by another number.

[0087] It can be seen from the data migration method of an embodiment of the present invention that the data table to be migrated is obtained from the source database according to the data source name; the target data is extracted from the data table to be migrated based on the table field; the table field is mapped to the target field, and the type of the target data is converted to the target type to generate the migration data; the migration data is stored in the target database. Therefore, the technical problems of low migration efficiency and the need for a lot of energy are overcome; and the migrated data also needs secondary processing to meet the needs, thereby achieving the technical effect of being able to efficiently migrate massive data; and in the process of migrating massive data, the massive data is subjected to secondary processing to be processed into data that meets the needs of analysis or search.

[0088] like Figure 2 As shown, the data migration method of the embodiment of the present invention can refer to the implementation process as follows:

[0089] Step S201: using Spark to obtain a data table to be migrated from a Hive data warehouse according to at least one database name and / or data table name;

[0090] Step S202: extracting target data from the to-be-migrated data table based on the table fields and the values ​​of the table fields;

[0091] Step S203: Map the table fields to the target fields, and convert the type of the target data to the target type, to generate migration data;

[0092] Step S204: querying the target database whether there is a storage data table storing migration data;

[0093] If it does not exist, execute step S205; if it does exist, execute step S206.

[0094] Step S205: creating a storage data table based on the target field;

[0095] Step S206: performing a modulo operation on the hash value of the data source name and the date of the migration data to obtain a modulo remainder; matching a target subcluster from subclusters of the target database based on the modulo remainder; and inserting the migration data into the target subcluster.

[0096] like Figure 3 As shown, based on the architecture shown in the figure, the implementation of the data migration method of the embodiment of the present invention is mainly divided into the following parts: using Spark to read data from hive, filtering data, remapping data fields and converting types, and using the JDBC API provided by ClickHouse to batch insert data.

[0097] like Figure 4 As shown, when applying the data migration method of the embodiment of the present invention, the following process can be followed:

[0098] First, the reading and writing of data depends on Spark. SparkSQL can be used to read data in the Hive data warehouse through the JDBC API to form a DateFrame:

[0099] Here, you can automatically read data through the configuration of the script. Specifically, specify the database name (DataBase Name) and / or table name (TableName) that needs to be mapped in the data source and target source;

[0100] Secondly, Spark SQL is used to operate the read data set (i.e., the data table to be migrated). In the business, it is necessary to extract the fields required by the business (i.e., table fields), and it is necessary to filter out some useless fields in the Hive data warehouse. At the same time, the field names in the Hive data warehouse can be mapped. For example, in the business, it is necessary to facilitate the understanding of the calculation of unique visitors (UV). In ClickHouse, the field name is UV, but in the Hive warehouse, it is identified by the browser unique identifier browseUniqId. However, this problem can be solved by mapping the fields:

[0101] Here, you can also automatically filter data through the configuration of the script. Specifically, perform field mapping and type conversion configuration, specify the required Hive table field name and the corresponding field name in ClickHouse, and set the conditional filtering field. For example, only the data with the business mode of self-operation is required, and the data with the business mode of POP is not required.

[0102] Then, a new dataset (including all migrated data) is formed by operating the dataset, including mapping, filtering, and type conversion of fields, i.e., the data required by the business.

[0103] Finally, create a database and table in ClickHouse (that is, create a storage data table) based on the mapped target field. Then, take the hash value of the table name in ClickHouse and calculate it with the date in each piece of data. Then, modulo the obtained value with the number of shards in ClickHouse. Based on this shard number, it is determined that the data on the same day falls on the same shard. The purpose of this is to ensure the data query rate.

[0104] In ClickHouse, MergeTree is the most ideal storage engine provided by ClickHouse database. According to the experience in use, MergeTree is the most reasonable to cooperate with the distributed engine (Distributed). Distributed does not store real data, but is used for distributed writing and querying, and is used in conjunction with other engines. Therefore, there is a reasonable cluster topology such as Figure 6 As shown:

[0105] According to the original data distribution, the data written to the ClickHouse cluster is written in a polling manner. For example, if there are 100 million data records per day, each batch is written to each shard of ClickHouse separately, and the data of the same day is distributed to each shard of the cluster. The daily data is statistically analyzed based on the business. After the data on different shards are aggregated, the data on each shard will be aggregated again on the requesting client. In order to aggregate the data at one time and reduce the query response time, when writing data to ClickHouse, the corresponding algorithm is used to make all the data of the same day written to the same shard.

[0106] In order to further explain the technical idea of ​​the present invention, the technical solution of the present invention is now described in combination with specific application scenarios.

[0107] Assume that based on business needs, data needs to be migrated from Hive to ClickHouse every day. A big data platform provides a task scheduling function. Package the data migration program and publish it to the big data platform, and then publish the script for scheduling the migration program to the big data platform. When deploying tasks, you can set the time and rules for the daily execution of the scheduling script. Standardize the migration of data to ClickHouse and implement scheduled data migration every day.

[0108] Figure 6 4 is a schematic diagram of main modules of a data migration device according to an embodiment of the present invention.

[0109] like Figure 6 As shown, the data migration device 600 according to the embodiment of the present invention includes: an acquisition module 601 , an extraction module 602 , a generation module 603 and a storage module 604 .

[0110] in,

[0111] An acquisition module 601 is used to acquire a data table to be migrated from a source database according to a data source name;

[0112] An extraction module 602 is used to extract target data from the to-be-migrated data table based on table fields;

[0113] A generating module 603 is used to map the table fields to target fields and convert the type of the target data to the target type to generate migration data;

[0114] The storage module 604 is used to store the migration data in a target database.

[0115] In the embodiment of the present invention, the extraction module 602 is further used for:

[0116] Extracting intermediate data from the to-be-migrated data table based on table fields;

[0117] The target data is filtered out from the intermediate data using the value of the table field.

[0118] In the embodiment of the present invention, the storage module 604 is further used for:

[0119] Based on the data source name, query whether there is a storage data table storing the migration data in the target database; wherein the storage data table corresponds to the data source name;

[0120] If it does not exist, create the storage data table based on the target field; and insert the migration data into the storage data table of the sub-cluster of the target database; wherein the storage data table is distributed and stored in each sub-cluster of the target database;

[0121] If so, insert the migration data into the storage data table of the sub-cluster of the target database.

[0122] In the embodiment of the present invention, the storage module 604 is further used for:

[0123] Performing a modulo operation on the hash value of the data source name and the date of the migration data to obtain a modulo remainder;

[0124] matching a target subcluster from the subcluster of the target database based on the modulus;

[0125] Insert the migration data into the target subcluster.

[0126] In addition, the source database is a Hive data warehouse, and the target database is ClickHouse.

[0127] In the embodiment of the present invention, the acquisition module 601 is further used for:

[0128] Spark is used to obtain a data table to be migrated from a Hive data warehouse according to at least one database name and / or data table name; wherein the data table to be migrated is a matrix data table having a row index and a column index.

[0129] It can be seen from the data migration device according to the embodiment of the present invention that the data table to be migrated is obtained from the source database according to the data source name; the target data is extracted from the data table to be migrated based on the table field; the table field is mapped to the target field, and the type of the target data is converted to the target type to generate the migration data; the migration data is stored in the target database. Therefore, the technical problems of low migration efficiency and the need for a lot of energy are overcome; and the migrated data also needs secondary processing to meet the needs, thereby achieving the technical effect of being able to efficiently migrate massive data; and in the process of migrating massive data, the massive data is subjected to secondary processing to be processed into data that meets the needs of analysis or search.

[0130] Figure 7 An exemplary system architecture 700 is shown to which the method or apparatus for data migration according to an embodiment of the present invention may be applied.

[0131] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, 703, network 704 and server 705. Network 704 is used to provide a medium for communication links between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0132] Users can use terminal devices 701, 702, and 703 to interact with server 705 through network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0133] The terminal devices 701 , 702 , and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0134] The server 705 may be a server that provides various services, such as a backend management server that provides support for shopping websites browsed by users using the terminal devices 701, 702, and 703. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information and product information) to the terminal device.

[0135] It should be noted that the data migration method provided in the embodiment of the present invention is generally executed by the server 705 , and accordingly, the data migration device is generally set in the server 705 .

[0136] It should be understood that Figure 7The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0137] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 of a terminal device suitable for implementing an embodiment of the present invention. Figure 8 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0138] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0139] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0140] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are executed.

[0141] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0142] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0143] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes an acquisition module, an extraction module, a generation module and a storage module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the acquisition module may also be described as "a module for acquiring a data table to be migrated from a source database according to a data source name".

[0144] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: step S101: obtaining a data table to be migrated from a source database according to a data source name; step S102: extracting target data from the data table to be migrated based on table fields; step S103: mapping table fields to target fields, and converting the type of target data to target type to generate migration data; step S104: storing the migration data in the target database.

[0145] According to the technical solution of the embodiment of the present invention, the data table to be migrated is obtained from the source database according to the data source name; the target data is extracted from the data table to be migrated based on the table field; the table field is mapped to the target field, and the type of the target data is converted to the target type to generate the migration data; the migration data is stored in the target database. Therefore, the technical problems of low migration efficiency and the need for a lot of energy and the need for secondary processing of the migrated data to meet the needs are overcome, thereby achieving the technical effect of being able to efficiently migrate massive data; and in the process of migrating massive data, the massive data is subjected to secondary processing to be processed into data that meets the needs of analysis or search.

[0146] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for data migration, characterized in that: include: Get the data table to be migrated from the source database according to the data source name; Extracting target data from the to-be-migrated data table based on table fields; Mapping the table fields to target fields and converting the type of the target data to the target type to generate migration data; Storing the migration data into a target database includes: querying whether a storage data table storing the migration data exists in the target database based on the data source name; wherein the storage data table corresponds to the data source name; if it exists, inserting the migration data into the storage data table of the sub-cluster of the target database evenly according to the date; Each of the sub-clusters corresponds to a plurality of shards, and the data of the same day is stored on one shard; the migration data is inserted into the storage data table of the sub-cluster of the target database, including: performing a modulus operation on the hash value of the data source name and the date of the migration data to obtain a modulus remainder; matching a target sub-cluster from the sub-clusters of the target database based on the modulus remainder; and inserting the migration data into the target sub-cluster evenly according to the date.

2. The method according to claim 1, characterized in that: Extracting target data from the to-be-migrated data table based on table fields includes: Extracting intermediate data from the to-be-migrated data table based on table fields; The target data is filtered out from the intermediate data using the value of the table field.

3. The method according to claim 1, characterized in that: Storing the migration data in a target database further includes: If the target database does not have a storage data table storing the migration data, create the storage data table based on the target field; and insert the migration data into the storage data table of the subcluster of the target database; wherein the storage data table is distributed and stored in each subcluster of the target database.

4. The method according to claim 1, characterized in that: The source database is a Hive data warehouse, the target database is ClickHouse, and Get the data table to be migrated from the source database according to the data source name, including: Using Spark, a data table to be migrated is obtained from a Hive data warehouse according to at least one database name and / or data table name; wherein the data table to be migrated is a matrix data table having a row index and a column index.

5. A data migration device, characterized in that: include: The acquisition module is used to obtain the data table to be migrated from the source database according to the data source name; An extraction module, used for extracting target data from the to-be-migrated data table based on table fields; A generation module, used for mapping the table fields to target fields and converting the type of the target data to the target type to generate migration data; A storage module, used for storing the migration data in a target database, comprising: querying whether there is a storage data table storing the migration data in the target database based on the data source name; wherein the storage data table corresponds to the data source name; if it exists, inserting the migration data into the storage data table of the sub-cluster of the target database evenly according to the date; Each of the sub-clusters corresponds to a plurality of shards, and the data of the same day is stored on one shard; the migration data is inserted into the storage data table of the sub-cluster of the target database, including: performing a modulus operation on the hash value of the data source name and the date of the migration data to obtain a modulus remainder; matching a target sub-cluster from the sub-clusters of the target database based on the modulus remainder; and inserting the migration data into the target sub-cluster evenly according to the date.

6. The device according to claim 5, characterized in that The extraction module is also used for: Extracting intermediate data from the to-be-migrated data table based on table fields; The target data is filtered out from the intermediate data using the value of the table field.

7. The device according to claim 6, characterized in that The storage module is also used for: If the target database does not have a storage data table storing the migration data, create the storage data table based on the target field; and insert the migration data into the storage data table of the subcluster of the target database; wherein the storage data table is distributed and stored in each subcluster of the target database.

8. The device according to claim 5, characterized in that The source database is a Hive data warehouse, the target database is ClickHouse, and The acquisition module is also used for: Using Spark, a data table to be migrated is obtained from a Hive data warehouse according to at least one database name and / or data table name; wherein the data table to be migrated is a matrix data table having a row index and a column index.

9. An electronic device for data migration, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images