Data replication method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data replication method and device, electronic equipment and a storage medium, and belongs to the technical field of big data. The method comprises the steps that configuration information is obtained, the configuration information comprises data information of a source data table and an identifier of a target library, and the data information comprises a first table name and a table structure of the source data table; processing the table structure according to a first rule to obtain a first identifier; the first table name is matched with a second table name in a metadata record table to obtain a target record item, the metadata record table comprises a plurality of record items, each record item comprises the second table name and a second identifier, and the second identifier is obtained by processing a table structure corresponding to the second table name according to a first rule; if the second identifier of the target record item is different from the first identifier, creating a first target table for the source data table; and copying the data of the source data table into the first target table. By copying the data of the source data table into the first target table, the data synchronization of the source data table and the first target table is realized.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data replication method, device, electronic device and storage medium. Background Art

[0002] Hive and Impala are two commonly used tools in the field of big data. In the common solution of using Hive and Impala, Hive is used as a data warehouse for data storage and batch processing. Impala can be used for real-time query and interactive analysis. By executing queries directly on the Hadoop data node, faster query response time can be obtained. These two tools can work together to provide a comprehensive data analysis solution. Creating an Impala table and pointing the data file location to the Hive file is a common derivative solution, but this solution is prone to data consistency problems when multiple Impala tables operate on the same Hive. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to provide a data replication method, device, electronic device and storage medium, which can solve the problem of data inconsistency when multiple Impala tables operate on the same Hive.

[0004] To achieve the above object, a first aspect of an embodiment of the present application provides a data replication method, the method comprising:

[0005] Acquire configuration information, the configuration information including data information of a source data table and an identifier of a target database, the data information including a first table name and a table structure of the source data table;

[0006] Processing the table structure according to a first rule to obtain a first identifier;

[0007] Matching the first table name with the second table name in the metadata record table to obtain a target record item, wherein the metadata record table includes multiple record items, each record item includes a second table name and a second identifier, the second identifier is obtained by processing the table structure corresponding to the second table name according to the first rule, and the second table name in the target record item is the same as the first table name;

[0008] If the second identifier of the target record item is different from the first identifier, a first target table is created for the source data table, the first target table is a table in the target database, and the source data table and the first target table have the same table structure;

[0009] The data of the source data table is copied to the first target table.

[0010] In some embodiments, if the second identifier of the target record item is different from the first identifier, creating a first target table for the source data table includes:

[0011] When the second identifier of the target record item is different from the first identifier and the target record item satisfies a first condition, the first target table and a data partition corresponding to the first target table are created for the source data table, and the table structure of the source data table is the same as the table structure of the first target table, wherein the first condition includes: the target record item does not include the identifier of the table in the target library; or the target record item includes the identifier of the table in the target library;

[0012] Recording the table name of the first target table and the partition information of the data partition corresponding to the first target table in the metadata record table;

[0013] The step of copying the data in the source data table to the first target table includes:

[0014] The data in the data partition corresponding to the source data table is copied to the data partition corresponding to the first target table.

[0015] In some embodiments, recording the table name of the first target table and the partition information of the data partition corresponding to the first target table in the metadata record table includes:

[0016] If the target record item does not include the identifier of the table in the target library, recording the table name of the first target table and the partition information of the data partition corresponding to the first target table in the target record item;

[0017] If the target record item includes the identifier of the table in the target library, a new record item is added to the metadata record table, and the newly added record item includes the first table name, the first identifier, the table name of the first target table, and the partition information of the data partition corresponding to the first target table.

[0018] In some embodiments, when the target record item does not include an identifier of a table in the target library, the table name of the first target table is the same as the first table name;

[0019] In the case where the target record item includes an identifier of a table in the target library, the table name of the first target table adopts a temporary table name, and the temporary table name is determined according to the first table name and the time when the first target table is created.

[0020] In some embodiments, after copying the data of the source data table to the first target table, the method further includes:

[0021] In a case where the target record item does not include an identifier of a table in the target library, recording partition information of the first target table in a partition information table;

[0022] In the case where the target record item includes the identifier of the table in the target library, the partition information of the old table in the partition information table is deleted, and the target record item and the data stored in the data partition corresponding to the old table are deleted, the temporary table name in the newly added record item in the metadata record table is modified to the first table name, and the partition information of the first target table is recorded in the partition information table;

[0023] The partition information table includes a plurality of information items, each of which is used to describe the storage address and / or stored data of a data partition corresponding to a table.

[0024] In some embodiments, the target record item further includes partition information of a data partition corresponding to a second target table, where the second target table is a table in the target library;

[0025] After matching the first table name with the second table name in the metadata record table to obtain the target record item, the method further includes:

[0026] If the second identifier of the target record item is the same as the first identifier, obtaining partition information of the data partition corresponding to the second target table from the target record item;

[0027] The data in the data partition corresponding to the source data table is copied to the data partition corresponding to the second target table.

[0028] In some embodiments, the source data table is a hive table, and the first target table is an impala table.

[0029] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a data replication device, the device comprising:

[0030] A first acquisition module is used to acquire configuration information, wherein the configuration information includes data information of a source data table and an identifier of a target database, wherein the data information includes a first table name and a table structure of the source data table;

[0031] A second acquisition module, configured to process the table structure according to a first rule to obtain a first identifier;

[0032] a matching module, configured to match the first table name with a second table name in the metadata record table to obtain a target record item, wherein the metadata record table includes a plurality of record items, each record item includes a second table name and a second identifier, the second identifier is obtained by processing a table structure corresponding to the second table name according to the first rule, and the second table name in the target record item is the same as the first table name;

[0033] a creation module, configured to create a first target table for the source data table if the second identifier of the target record item is different from the first identifier, wherein the first target table is a table in the target library, and the source data table and the first target table have the same table structure;

[0034] A data replication module is used to replicate the data in the source data table to the first target table.

[0035] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0036] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0037] The data replication method, device, electronic device and storage medium proposed in the present application obtain configuration information, wherein the configuration information includes data information of a source data table and an identifier of a target library, wherein the data information includes a first table name and a table structure of the source data table; the table structure is processed according to a first rule to obtain a first identifier; the first table name is matched with a second table name in the metadata record table to obtain a target record item, wherein the metadata record table includes multiple record items, each record item includes a second table name and a second identifier, wherein the second identifier is obtained by processing a table structure corresponding to the second table name according to the first rule, and the second table name in the target record item is the same as the first table name; if the second identifier of the target record item is different from the first identifier, a first target table is created for the source data table, wherein the first target table is a table in the target library, and the source data table and the first target table have the same table structure; and the data of the source data table is copied to the first target table. Through the above process, when the second identifier in the target record item is different from the first identifier (that is, when the table structure of the source data table changes), the data of the source data table can be copied to the first target table to achieve data synchronization between the source data table and the first target table. When the above solution is applied to the big data field using Hive and Impala, the problem of data inconsistency between the Hive table and the Impala table can be avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of a data replication method provided by an embodiment of the present application;

[0039] Figure 2 The embodiment of this application provides Figure 1 A flow chart of step S104 in FIG.

[0040] Figure 3 The embodiment of this application provides Figure 2 A flowchart of step S1042 in FIG.

[0041] Figure 4 is another flow chart of the data replication method provided by an embodiment of the present application;

[0042] Figure 5 is another flow chart of the data replication method provided in the embodiment of the present application;

[0043] Figure 6 is a flow chart of a data replication method provided by an embodiment of the present application;

[0044] Figure 7 is a structural schematic diagram of a data replication device provided in an embodiment of the present application;

[0045] Figure 8It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0049] First, some nouns involved in this application are analyzed:

[0050] Hive: Hive is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. It is a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop. The hive data warehouse tool can map structured data files to a database table, and provides SQL query functions, which can convert SQL statements into MapReduce tasks for execution. The advantage of hive is that it has a low learning cost, and can implement fast MapReduce statistics through similar SQL statements, making MapReduce simpler without having to develop a dedicated MapReduce application. Hive is very suitable for statistical analysis of data warehouses;

[0051] Hive table: table in hive;

[0052] Impala: Impala is a massively parallel processing (MPP) SQL query engine for processing large amounts of data stored in Hadoop clusters. Impala is a new query system developed by Cloudera, which provides SQL semantics and can query big data stored in Hadoop's distributed file system (Hadoop Distributed File System, HDFS);

[0053] Impala table: a table in Impala.

[0054] Hive and Impala are two commonly used tools in the field of big data. In the common scheme of using Hive and Impala, Hive is used as a data warehouse for data storage and batch processing. Impala can be used for real-time query and interactive analysis. By executing queries directly on the Hadoop data node, faster query response time can be obtained. These two tools can work together to provide a comprehensive data analysis solution. Creating an Impala external table and pointing the data file location to the Hive file is a common derivative solution. However, this solution is prone to data consistency problems when multiple Impala tables operate on the same Hive. When the Hive field information or metadata information changes, it cannot be synchronized to Impala. It is often necessary to manually create a new Impala table based on the metadata changes. In other words, the use of existing Hive tables and Impala tables is prone to data inconsistency problems.

[0055] To solve the above problems, the embodiments of the present application provide a data replication method, device, electronic device and storage medium.

[0056] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0057] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0058] Figure 1 is a flow chart of a data replication method provided in an embodiment of the present application, Figure 1 The method may include but is not limited to steps S101 to S105, wherein:

[0059] Step S101, obtaining configuration information, wherein the configuration information includes data information of a source data table and an identifier of a target database, wherein the data information includes a first table name and a table structure of the source data table.

[0060] The table structure may include the field name and field type of the source data table. The source data table may be a table in the source database, and the source database and the target database are different databases.

[0061] In the embodiment of the present application, the source data table may be a table in hive, and the first target table may be a table in impala.

[0062] Step S102: Process the table structure according to a first rule to obtain a first identifier.

[0063] The first rule can be set according to actual conditions and is not limited here. For example, the first rule can be to process the table structure using the Message-Digest Algorithm (MD5) to obtain a hash value after processing, which is called the first identifier. Specifically, the field names and field types of the table structure can be concatenated to obtain a concatenated string, and the concatenated string can be processed using MD5 to obtain the first identifier.

[0064] The first rule may also be to process the table structure and the partition information of the data partition of the source data table using MD5, wherein the data partition of the source data table refers to the storage area for storing the data of the source data table, and the partition information is used to describe the storage address and / or stored data of the data partition. For example, the field names, field types and partition information of the table structure are concatenated to obtain a concatenated string, and the concatenated string is processed using MD5 to obtain a first identifier.

[0065] In the above, MD5 is a widely used cryptographic hash function that can generate a 128-bit (16-byte) hash value (also called hash value) to ensure the integrity and consistency of information transmission. In this embodiment, MD5 can also be replaced by other algorithms for generating hash values.

[0066] It should be noted that the first rule at least needs to be processed based on the table structure of the source data table, so as to facilitate subsequent comparison of the generated hash value (ie, the first identifier) ​​to determine whether the table structure of the source data table has changed.

[0067] Step S103: Match the first table name with the second table name in the metadata record table to obtain a target record item. The metadata record table includes multiple record items, each record item includes a second table name and a second identifier, and the second identifier is obtained by processing the table structure corresponding to the second table name according to the first rule. The second table name in the target record item is the same as the first table name.

[0068] For ease of description, the table name in the record item is referred to as the second table name, and the identifier obtained by processing the table structure corresponding to the second table name according to the first rule is referred to as the second identifier. It should be noted that the record item may also include the identifier of the table in the target library (for example, the identifier may be the table name, and the table name may be the same as the second table name in the record item), but not every record item includes the identifier of the table in the target library. For example, if the hive table has a corresponding impala table, the record item may include the table name of the hive table, the second identifier, the table name of the impala table, and the partition information of the data partition of the impala table.

[0069] According to the first table name, a search is performed in the metadata record table to find a record item where a second table name identical to the first table name is located. The record item is called a target record item.

[0070] Step S104: if the second identifier of the target record item is different from the first identifier, a first target table is created for the source data table, the first target table is a table in the target library, and the source data table and the first target table have the same table structure.

[0071] After finding the target record item, match the second identifier in the target record item with the first identifier. If the two are the same, it means that the table structure of the source data table is the same as the record in the metadata record table, and the table structure of the source data table has not changed; if the two are different, it means that the table structure of the source data table is different from the record in the metadata record table, and the table structure of the source data table has changed. It is necessary to create a first target table for the source data table. For example, the source data table is a hive table. When it is determined that the second identifier in the target record item is different from the first identifier, an impala table (i.e., the first target table) is created for the hive table. The table structure of the created impala table is the same as the table structure of the hive table, including the same field names and field types.

[0072] Step S105: copy the data in the source data table to the first target table.

[0073] The source data table and the first target table are based on different HDFS, which can avoid coupling of data in the source data table and data in the first target table. The data in the source data table can be business data, transaction data, payment data, etc. in the financial field, or data in other fields, which is not limited here.

[0074] Specifically, the data of the source data table is stored in the data partition of the source data table, and the first target table also has a corresponding data partition. When copying data, the data in the data partition of the source data table is copied to the data partition of the first target table to realize data copying of the source data table.

[0075] In steps S101 to S105 shown in the embodiment of the present application, configuration information is obtained, wherein the configuration information includes data information of a source data table and an identifier of a target library, wherein the data information includes a first table name and a table structure of the source data table; the table structure is processed according to a first rule to obtain a first identifier; the first table name is matched with a second table name in the metadata record table to obtain a target record item, wherein the metadata record table includes a plurality of record items, each record item includes a second table name and a second identifier, wherein the second identifier is obtained by processing a table structure corresponding to the second table name according to the first rule, and the second table name in the target record item is the same as the first table name; if the second identifier of the target record item is different from the first identifier, a first target table is created for the source data table, wherein the first target table is a table in the target library, and the source data table and the first target table have the same table structure; and the data of the source data table is copied to the first target table. Through the above process, when the second identifier in the target record item is different from the first identifier (that is, when the table structure of the source data table changes), the data of the source data table can be copied to the first target table to achieve data synchronization between the source data table and the first target table. When the above solution is applied to the big data field using Hive and Impala, the problem of data inconsistency between the Hive table and the Impala table can be avoided.

[0076] See also Figure 2 In some embodiments, step S104 may include but is not limited to steps S1041 to S1042:

[0077] Step S1041, when the second identifier of the target record item is different from the first identifier and the target record item satisfies a first condition, creating the first target table and the data partition corresponding to the first target table for the source data table, wherein the first condition includes: the target record item does not include the identifier of the table in the target library; or the target record item includes the identifier of the table in the target library, and the table structure of the source data table is the same as the table structure of the first target table.

[0078] If a corresponding target table has been created for the first table name corresponding to the source data table before, the identifier of the created target table (the identifier can also be the table name) will be recorded in the target record item. In this case, the target record item includes the identifier of the table in the target library; if a corresponding target table has not been created for the first table name corresponding to the source data table before, the target table corresponding to the first table name will not be recorded in the target record item. In this case, the target record item does not include the identifier of the table in the target library.

[0079] When creating the first target table, the table structure of the first target table is created according to the table structure of the source data table, and the table name of the first target table can be set to the first table name, so that the user can use the source data table and the first target table located in different databases according to the same table name.

[0080] The data partition corresponding to the first target table is used to store the data of the first target table.

[0081] Step S1042: Record the table name of the first target table and the partition information of the data partition corresponding to the first target table in the metadata record table.

[0082] The partition information of the data partition corresponding to the first target table is used to describe the storage address and / or stored data of the data partition. The partition information of the data partition corresponding to the first target table may include an identifier of the data partition, and the storage address of the data partition can be determined according to the identifier.

[0083] Correspondingly, step S105, copying the data of the source data table to the first target table, includes:

[0084] Step S1051, copying the data in the data partition corresponding to the source data table to the data partition corresponding to the first target table.

[0085] In steps S1041 to S1043 shown in this embodiment, when the second identifier of the target record item is different from the first identifier and the target record item satisfies the first condition, it indicates that the table structure of the source data table has changed. Regardless of whether a corresponding table has been created for the source data table before, in this case, it is necessary to create a first target table for the source data table. The first target table is the table corresponding to the source data table in the target library. Then, the data in the data partition corresponding to the source data table is copied to the data partition corresponding to the first target table to achieve data synchronization between the source data table and the first target table. When the above solution is applied to the big data field using Hive and Impala, data inconsistency problems between Hive tables and Impala tables can be avoided.

[0086] See also Figure 3 In some embodiments, step S1042, recording the table name of the first target table and the partition information of the data partition corresponding to the first target table in the metadata record table, may include but is not limited to steps S10421 to S10422:

[0087] Step S10421: If the target record item does not include the identifier of the table in the target library, the table name of the first target table and the partition information of the data partition corresponding to the first target table are recorded in the target record item.

[0088] The target record item does not include the identifier of the table in the target library, indicating that the corresponding table has not been created for the source data table before. In this case, after creating the corresponding first target table for the source data table, the table name of the first target table and the partition information of the data partition corresponding to the first target table are recorded in the target record item. When the target record item is subsequently found in the metadata record table again according to the first table name, the historical record of creating the first target table for the first table name can be obtained, which is beneficial to data consistency.

[0089] Step S10422, if the target record item includes the identifier of the table in the target library, then a new record item is added to the metadata record table, and the newly added record item includes the first table name, the first identifier, the table name of the first target table, and the partition information of the data partition corresponding to the first target table.

[0090] The target record item includes the identifier of the table in the target library, indicating that a corresponding table has been created for the source data table before. In this case, since the table structure of the source data table has changed, the previously created table can no longer be used, and it is necessary to create a corresponding first target table for the source data table. After that, a new record item is added to the metadata record table. The newly added record item includes the first table name, the first identifier, the table name of the first target table, and the partition information of the data partition corresponding to the first target table. When the metadata record table is searched again according to the first table name, the historical record of creating the first target table for the first table name can be obtained, which is conducive to data consistency.

[0091] In some embodiments, when the target record item does not include an identifier of a table in the target library, the table name of the first target table is the same as the first table name;

[0092] In the case where the target record item includes an identifier of a table in the target library, the table name of the first target table adopts a temporary table name, and the temporary table name is determined according to the first table name and the time when the first target table is created.

[0093] In the above, the target record item does not include the identifier of the table in the target library, indicating that the corresponding table in the target library has not been created for the first table name before. In this case, the first target table will be created directly, and the table name of the first target table will be set to be the same as the first table name. In this way, the user can use the source data table and the first target table located in different databases according to the same table name.

[0094] The target record item includes the identifier of the table in the target database, indicating that a corresponding table in the target database has been created for the first table name. In this case, since the table structure of the source data table has changed, the previously created table cannot be used again, and a corresponding table needs to be created again for the source data table. In order to distinguish it from the previously created table (the table name of the previously created table is the same as the first table name), the table name of the first target table created here adopts a temporary table name, which is determined according to the first table name and the time when the first target table is created. For example, the first table name is "User Basic Table", and the time when the first target table is created is "8:12:56". The first table name and the time when the first target table is created are concatenated to obtain a temporary table name: "User Basic Table + 8:12:56". Later, after deleting the target record item (i.e., deleting the relevant information of the previously created table in the metadata record table), the temporary table name can be modified to the first table name. In this way, the user can use the source data table and the first target table located in different databases according to the same table name.

[0095] See also Figure 4 In step S105, after copying the data of the source data table to the first target table, the method further includes step S106 or step S106':

[0096] Step S106, when the target record item does not include the identifier of the table in the target library, recording the partition information of the first target table in the partition information table;

[0097] Step S106', in the case where the target record item includes the identifier of the table in the target library, the partition information of the old table in the partition information table is deleted, the old table is the table in the target library included in the target record item, and the target record item is deleted (the table is the corresponding table created for the first table name last time, and when deleting, the definition of the table, such as the table name, table structure, etc.) and the data stored in the data partition corresponding to the old table are deleted, and the temporary table name in the newly added record item in the metadata record table is modified to the first table name, and the partition information of the first target table is recorded in the partition information table;

[0098] The partition information table includes a plurality of information items, each of which is used to describe the storage address and / or stored data of a data partition corresponding to a table.

[0099] Among them, each information item may include a unique identifier (for example, the unique identifier may be the database name where the table is located + the table name) and the partition information of the table, and each partition information is used to record the storage address of each data partition of the corresponding table, and / or the information of the data stored at each storage address, such as data size, storage time, etc.

[0100] The partition information table also records the partition information of the source data table, and the data of the source data table can be obtained based on the partition information.

[0101] In the above, the partition information of the first target table is recorded in the partition information table. When the data of the first target table is subsequently acquired, the data storage address of the first target table can be determined according to the record in the partition information table, thereby retrieving data according to the data storage address.

[0102] See also Figure 5 In some embodiments of the present application, the target record item further includes partition information of a data partition corresponding to a second target table, and the second target table is a table in the target library. Accordingly, in step S103, after matching the first table name with the second table name in the metadata record table to obtain the target record item, the method further includes steps S107 and S108:

[0103] Step S107: if the second identifier of the target record item is the same as the first identifier, obtaining partition information of the data partition corresponding to the second target table from the target record item;

[0104] Step S108: copy the data in the data partition corresponding to the source data table to the data partition corresponding to the second target table.

[0105] In the above, if the second identifier of the target record item is the same as the first identifier, it means that the table structure of the source data table has not changed, and the data can be copied directly. The data partition corresponding to the source data table can be determined according to the partition information table to retrieve the data, and the partition information of the data partition corresponding to the second target table is obtained from the target record item. Then, the data of the source data table is copied to the data partition corresponding to the second target table, so as to realize data synchronization between the source data table and the second target table.

[0106] The following uses the example of the data replication method provided in the embodiment of the present application to illustrate the data replication method provided in the embodiment of the present application from the derivative of the hive table to the impala table. Figure 6 FIG. 1 is a flow chart of a data replication method provided in an embodiment of the present application.

[0107] (1) Create the Spark and Hive execution environments, read the configuration information, and check whether the configuration information is correct, for example, whether the data source is Hive and whether the target database is Impala.

[0108] (2) Query the table structure of the hive table, parse out all field names, field types, and partition information, concatenate and sort them, and calculate an MD5 value (i.e., the first identifier). Query the MD5 value corresponding to the table name of the hive table from the metadata record table (table_meta_info) (i.e., the second identifier), and determine whether the two MD5 values ​​are the same:

[0109] 1) If the two are consistent, it means that the table structure of the Hive table has not changed, and there is no need to adjust the existing Impala table;

[0110] 2) If the two are inconsistent and the hive table has no corresponding impala table, write the current table metadata (such as the table name of the hive table) into the metadata record table, create the impala table and data partition corresponding to the hive table, and set the metadata change flag update_flag = true;

[0111] 3) If the two are inconsistent and there is an Impala table corresponding to the Hive table, it means that the table structure of the Hive table has changed. Create an Impala temporary table and data partition corresponding to the Hive (the temporary table is named in the form of Hive table + time suffix), and set the metadata change flag update_flag = true;

[0112] (3) Copy the partition file of the hive table to the partition directory corresponding to the impala table HDFS;

[0113] (4) Determine whether the metadata has been updated based on update_flag, including:

[0114] 1) If the metadata changes (update_flag = true), delete the historical partition synchronization records of Hive in the partition information table (table_partitions_info) (that is, the partition information of the old table), use the drop command to delete the old Impala table and data files, and then use the alterrename command to change the table name of the Impala temporary table to the formal table name (that is, the first table name), record the partition information corresponding to the Impala table in the partition information table, and refresh the metadata of the Impala table (that is, update the records in the metadata record table, and change the temporary table name recorded in the metadata record table to the first table name).

[0115] 2) If the metadata has not changed (update_flag = false), query the partition information table (table_partitions_info) to determine whether there is a record of the current partition of the current synchronization table (that is, the partition information of the impala table). If there is, it means that the metadata has not changed and the partition information of this table already exists, and no operation is required; if not, update the partition information corresponding to the impala table and refresh the metadata of the impala table.

[0116] In the above, the impala table and hive table are based on different HDFS respectively, avoiding the coupling of hive data and impala data, and supporting one hive data source to derive (i.e. copy) different impala tables; in addition, the scheme automatically detects the metadata change information of the hive table (i.e. detects whether the table structure and partition information of the hive table have changed) every time it is derived. In the case of changes, the corresponding impala table is recreated using the change information without manual intervention, which improves the processing efficiency of data consistency while ensuring data consistency.

[0117] See also Figure 7 The embodiment of the present application further provides a data replication device, which can implement the above data replication method. The data replication device 900 includes:

[0118] A first acquisition module 901 is used to acquire configuration information, wherein the configuration information includes data information of a source data table and an identifier of a target database, wherein the data information includes a first table name and a table structure of the source data table;

[0119] A second acquisition module 902, configured to process the table structure according to a first rule to obtain a first identifier;

[0120] a matching module 903, configured to match the first table name with a second table name in the metadata record table to obtain a target record item, wherein the metadata record table includes a plurality of record items, each record item includes a second table name and a second identifier, wherein the second identifier is obtained by processing a table structure corresponding to the second table name according to the first rule, and the second table name in the target record item is the same as the first table name;

[0121] A creation module 904 is configured to create a first target table for the source data table if the second identifier of the target record item is different from the first identifier, wherein the first target table is a table in the target database, and the source data table and the first target table have the same table structure;

[0122] The first data copying module 905 is used to copy the data of the source data table to the first target table.

[0123] In some embodiments, the creation module 904 includes:

[0124] a creation submodule, configured to create the first target table and a data partition corresponding to the first target table for the source data table if the second identifier of the target record item is different from the first identifier and the target record item satisfies a first condition, wherein the first condition includes: the target record item does not include the identifier of the table in the target library; or the target record item includes the identifier of the table in the target library, and the table structure of the source data table is the same as the table structure of the first target table;

[0125] A recording submodule, used for recording the table name of the first target table and the partition information of the data partition corresponding to the first target table in the metadata recording table;

[0126] The first data replication module 905 is specifically configured to replicate the data in the data partition corresponding to the source data table to the data partition corresponding to the first target table.

[0127] In some embodiments, the recording submodule includes:

[0128] A first recording unit, configured to record, in the target record item, the table name of the first target table and the partition information of the data partition corresponding to the first target table, if the target record item does not include the identifier of the table in the target library;

[0129] A second recording unit is used to add a new record item in the metadata record table if the target record item includes the identifier of the table in the target library, and the newly added record item includes the first table name, the first identifier, the table name of the first target table, and the partition information of the data partition corresponding to the first target table.

[0130] In some embodiments, when the target record item does not include an identifier of a table in the target library, the table name of the first target table is the same as the first table name;

[0131] In the case where the target record item includes an identifier of a table in the target library, the table name of the first target table adopts a temporary table name, and the temporary table name is determined according to the first table name and the time when the first target table is created.

[0132] In some embodiments, the data replication device 900 further includes:

[0133] A first processing module, configured to record the partition information of the first target table in a partition information table when the target record item does not include an identifier of the table in the target library;

[0134] a second processing module, configured to, when the target record item includes an identifier of a table in the target library, delete the partition information of the old table in the partition information table, delete the target record item and the data stored in the data partition corresponding to the old table, modify the temporary table name in the newly added record item in the metadata record table to the first table name, and record the partition information of the first target table in the partition information table, wherein the old table is the table in the target library included in the target record item;

[0135] The partition information table includes a plurality of information items, each of which is used to describe the storage address and / or stored data of a data partition corresponding to a table.

[0136] In some embodiments, the target record item further includes partition information of a data partition corresponding to a second target table, and the second target table is a table in the target library;

[0137] The data replication device 900 further includes:

[0138] A third acquisition module, configured to acquire partition information of the data partition corresponding to the second target table from the target record item if the second identifier of the target record item is the same as the first identifier;

[0139] The second data replication module is used to replicate the data in the data partition corresponding to the source data table to the data partition corresponding to the second target table.

[0140] In some embodiments, the source data table is a hive table, and the first target table is an impala table.

[0141] The above-mentioned data replication device 900 can implement the above-mentioned data replication method and can achieve the same technical effect. For details, please refer to the description in the data replication method embodiment, which will not be repeated here.

[0142] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above data replication method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0143] See also Figure 8 , Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application, wherein the electronic device includes:

[0144] The processor 1001 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0145] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the data replication method of the embodiments of this application;

[0146] Input / output interface 1003, used to implement information input and output;

[0147] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0148] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0149] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0150] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above-mentioned data copying method when executed by a processor.

[0151] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0152] The present application also provides a computer program product, including a computer program / instruction, which, when executed by one or more processors, implements the steps of the data replication method of any embodiment of the present application.

[0153] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0154] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0155] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0157] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0158] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0159] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0160] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0161] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0162] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0163] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A data replication method, characterized in that: The method comprises: Acquire configuration information, the configuration information including data information of a source data table and an identifier of a target database, the data information including a first table name and a table structure of the source data table; Processing the table structure according to a first rule to obtain a first identifier; Matching the first table name with a second table name in a metadata record table to obtain a target record item, wherein the metadata record table includes multiple record items, each record item includes a second table name and a second identifier, the second identifier is obtained by processing a table structure corresponding to the second table name according to the first rule, and the second table name in the target record item is the same as the first table name; If the second identifier of the target record item is different from the first identifier, a first target table is created for the source data table, the first target table is a table in the target database, and the source data table and the first target table have the same table structure; The data of the source data table is copied to the first target table.

2. The data replication method according to claim 1, characterized in that: If the second identifier of the target record item is different from the first identifier, creating a first target table for the source data table includes: When the second identifier of the target record item is different from the first identifier and the target record item satisfies a first condition, the first target table and a data partition corresponding to the first target table are created for the source data table, and the table structure of the source data table is the same as the table structure of the first target table, wherein the first condition includes: the target record item does not include the identifier of the table in the target library; or the target record item includes the identifier of the table in the target library; Recording the table name of the first target table and the partition information of the data partition corresponding to the first target table in the metadata record table; The step of copying the data in the source data table to the first target table includes: The data in the data partition corresponding to the source data table is copied to the data partition corresponding to the first target table.

3. The data replication method according to claim 2, characterized in that: The step of recording the table name of the first target table and the partition information of the data partition corresponding to the first target table in the metadata record table includes: If the target record item does not include the identifier of the table in the target library, recording the table name of the first target table and the partition information of the data partition corresponding to the first target table in the target record item; If the target record item includes the identifier of the table in the target library, a new record item is added to the metadata record table, and the newly added record item includes the first table name, the first identifier, the table name of the first target table, and the partition information of the data partition corresponding to the first target table.

4. The data replication method according to claim 3, characterized in that: In the case where the target record item does not include an identifier of a table in the target database, the table name of the first target table is the same as the first table name; In the case where the target record item includes an identifier of a table in the target library, the table name of the first target table adopts a temporary table name, and the temporary table name is determined according to the first table name and the time when the first target table is created.

5. The data replication method according to claim 4, characterized in that: After copying the data in the source data table to the first target table, the method further includes: In a case where the target record item does not include an identifier of a table in the target library, recording partition information of the first target table in a partition information table; In the case where the target record item includes an identifier of a table in the target library, the partition information of the old table in the partition information table is deleted, and the data stored in the data partition corresponding to the target record item and the old table is deleted, the temporary table name in the newly added record item in the metadata record table is modified to the first table name, and the partition information of the first target table is recorded in the partition information table, where the old table is the table in the target library included in the target record item; The partition information table includes a plurality of information items, each of which is used to describe the storage address and / or stored data of a data partition corresponding to a table.

6. The data replication method according to claim 1, characterized in that: The target record item also includes partition information of a data partition corresponding to a second target table, where the second target table is a table in the target library; After matching the first table name with the second table name in the metadata record table to obtain the target record item, the method further includes: If the second identifier of the target record item is the same as the first identifier, obtaining partition information of the data partition corresponding to the second target table from the target record item; The data in the data partition corresponding to the source data table is copied to the data partition corresponding to the second target table.

7. The data replication method according to any one of claims 1 to 6, characterized in that: The source data table is a hive table, and the first target table is an impala table.

8. A data replication device, characterized in that: The device comprises: A first acquisition module is used to acquire configuration information, wherein the configuration information includes data information of a source data table and an identifier of a target database, wherein the data information includes a first table name and a table structure of the source data table; A second acquisition module, configured to process the table structure according to a first rule to obtain a first identifier; a matching module, configured to match the first table name with a second table name in a metadata record table to obtain a target record item, wherein the metadata record table includes a plurality of record items, each record item includes a second table name and a second identifier, the second identifier is obtained by processing a table structure corresponding to the second table name according to the first rule, and the second table name in the target record item is the same as the first table name; a creation module, configured to create a first target table for the source data table if the second identifier of the target record item is different from the first identifier, wherein the first target table is a table in the target library, and the source data table and the first target table have the same table structure; A data replication module is used to replicate the data in the source data table to the first target table.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the data copying method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data replication method according to any one of claims 1 to 7 is implemented.

Images