Method, device and electronic equipment for synchronizing hive data tables between clusters

Abstract

This application discloses a method, apparatus, and electronic device for synchronizing Hive data tables between clusters. Relating to the field of big data technology, the method includes: acquiring source cluster information, wherein the source cluster information includes at least: the name of the source cluster, the name of the Hive database in the source cluster, and the name of the Hive data table to be synchronized in the Hive database of the source cluster; determining target information of the Hive data table to be synchronized based on the source cluster information, wherein the target information includes at least: table structure information and authorization information, wherein the authorization information includes at least: the authorized object to which the permission belongs and the permission name; processing the target information to obtain processed target information; and synchronizing the processed target information to the target Hive data table in the Hive database of the target cluster. This application solves the problem of low synchronization efficiency of Hive tables across clusters in related technologies.

Description

Technical Field

[0001] This application relates to the field of big data technology, and more specifically, to a method, apparatus, and electronic device for synchronizing Hive data tables between clusters. Background Technology

[0002] In Hadoop clusters, user and role information is often managed separately, and user and role IDs stored in the backend of different clusters are not interchangeable. In Hive databases, the authorization for specific tables based on roles is related to the user and role IDs stored in the cluster backend, and the table structure and data are also stored separately. Therefore, when operating multiple clusters, the structure and authorization of the same Hive database table cannot be synchronized by simply copying the backend metadata.

[0003] Currently, the synchronization of Hive tables and their authorizations across different Hadoop clusters primarily involves manually collecting all table creation and authorization scripts, modifying them to adapt to the corresponding role IDs in the current cluster, and then synchronizing them on the target cluster via script execution or a backend interface. Because accumulating the full table structure and authorization scripts requires long-term execution, and given the different development paces of various applications and teams in a parallel, decentralized environment, version control requirements differ across environments, leading to significant management costs. Furthermore, the adaptation and modification work for user roles at each level after script collection requires substantial manpower, further reducing the efficiency of Hive table synchronization across clusters.

[0004] There is currently no effective solution to the aforementioned problems in the relevant technologies. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, and electronic device for synchronizing Hive data tables between clusters, in order to solve the problem of low synchronization efficiency of Hive tables across clusters in related technologies.

[0006] To achieve the above objectives, according to one aspect of this application, a method for synchronizing Hive data tables between clusters is provided. The method includes: obtaining source cluster information, wherein the source cluster information includes at least: the name of the source cluster, the name of the Hive database in the source cluster, and the name of the Hive data table to be synchronized in the Hive database of the source cluster; determining target information of the Hive data table to be synchronized based on the source cluster information, wherein the target information includes at least: table structure information and authorization information, wherein the authorization information includes at least: the authorized object to which the permission belongs and the permission name; processing the target information to obtain processed target information; and synchronizing the processed target information to the target Hive data table in the Hive database of the target cluster.

[0007] Furthermore, before synchronizing the processed target information to the target Hive data table in the target cluster's Hive database, the method further includes: in response to a query command input on the target interface, obtaining the database name of the target cluster's Hive database and the table name of the target Hive data table, wherein the query command is used to query information in the target cluster.

[0008] Furthermore, determining one of the target information for the Hive data table to be synchronized based on the source cluster information includes: obtaining the Hive interface information in the source cluster; connecting to the Hive database in the source cluster based on the Hive interface information; responding to the output instruction for table structure information, outputting the table structure information of the Hive database in the source cluster to obtain the output table structure information; and using the output table structure information as one of the target information.

[0009] Furthermore, determining one of the target information for the Hive data table to be synchronized based on the source cluster information includes: obtaining Hive metadata from the source cluster; identifying multiple metadata tables in the Hive metadata; extracting the database ID information of Hive in the source cluster from the multiple metadata tables; and determining one of the target information based on the database ID information of Hive in the source cluster.

[0010] Furthermore, determining one of the target information based on the ID information of the Hive database in the source cluster includes: extracting the ID information of the Hive data table to be synchronized based on the name of the Hive data table to be synchronized and the ID information of the Hive database in the source cluster; extracting the authorization information of the Hive data table to be synchronized from multiple authorization information tables based on the ID information of the Hive data table to be synchronized; and using the authorization information of the Hive data table to be synchronized as one of the target information.

[0011] Furthermore, the target information is processed to obtain the following: a target data table structure is created based on the table structure information, wherein the target data table structure is used to store the Hive data tables to be synchronized; when the authorized object of the permissions in the authorization information is a user group, the group role name in the user group is obtained, the role ID information of the same group in the target cluster is determined based on the group role name, the target authorization name corresponding to the user group is determined based on the role ID information, the authorization name in the source cluster is replaced based on the target authorization name to obtain the authorization information required by the target cluster; and the target data table structure and the authorization information required by the target cluster are used as the processed target information.

[0012] Furthermore, synchronizing the processed target information to the target Hive data table in the target cluster's Hive database includes: obtaining the target Hive interface information in the target cluster; connecting to the target cluster's Hive database based on the target Hive interface information; and synchronizing the processed target information to the target Hive data table in the target cluster's Hive database in response to the synchronization command.

[0013] To achieve the above objectives, according to another aspect of this application, a device for synchronizing Hive data tables between clusters is provided. The device includes: an acquisition unit for acquiring source cluster information, wherein the source cluster information includes at least: the name of the source cluster, the name of the Hive database in the source cluster, and the name of the Hive data table to be synchronized in the Hive database of the source cluster; a determination unit for determining target information of the Hive data table to be synchronized based on the source cluster information, wherein the target information includes at least: table structure information and authorization information, wherein the authorization information includes at least: the authorized object to which the permission belongs and the permission name; a processing unit for processing the target information to obtain processed target information; and a synchronization unit for synchronizing the processed target information to the target Hive data table in the Hive database of the target cluster.

[0014] This application employs the following steps: obtaining source cluster information, which includes at least: the name of the source cluster, the name of the Hive database in the source cluster, and the name of the Hive table to be synchronized in the Hive database of the source cluster; determining the target information of the Hive table to be synchronized based on the source cluster information, where the target information includes at least: table structure information and authorization information, where the authorization information includes at least: the authorized object to which the permission belongs and the permission name; processing the target information to obtain processed target information; and synchronizing the processed target information to the target Hive table in the Hive database of the target cluster. This application solves the problem of low synchronization efficiency of Hive tables across clusters in related technologies. By synchronizing the processed target information to the target Hive table in the Hive database of the target cluster, the synchronization efficiency of Hive tables across clusters is improved. Attached Figure Description

[0015] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0016] Figure 1 This is a flowchart of a method for synchronizing Hive data tables between clusters according to an embodiment of this application;

[0017] Figure 2 This is a schematic diagram of a device for synchronizing Hive data tables between clusters according to an embodiment of this application;

[0018] Figure 3 This is a schematic diagram of an electronic device for synchronizing Hive data tables between clusters according to an embodiment of this application. Detailed Implementation

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] Definitions of terms or nouns in this application:

[0023] Hadoop: A distributed system infrastructure developed by the Apache Software Foundation.

[0024] Hive: A data warehouse analysis system built on Hadoop that provides rich SQL query methods for analyzing large-scale data stored in Hadoop.

[0025] Synchronization: Enables the same content to be updated in a consistent manner across different environments.

[0026] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data with user permission or with full permission from all parties.

[0027] The present invention will now be described in conjunction with preferred implementation steps. Figure 1 This is a flowchart of a method for synchronizing Hive data tables between clusters according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps:

[0028] Step S101: Obtain source cluster information, wherein the source cluster information includes at least: the name of the source cluster, the name of the Hive database in the source cluster, and the name of the Hive data table to be synchronized in the Hive database in the source cluster.

[0029] Specifically, this application can obtain information from multiple user information databases of multiple Hadoop source clusters, retrieve the full list of roles in the cluster, and form a parameter table to record the cluster name. For example, the information recorded for the source cluster can be the name of the source cluster A, the name of the Hive database B in the source cluster A, and the name of the Hive data table to be synchronized in the Hive database B.

[0030] Step S102: Determine the target information of the Hive data table to be synchronized based on the source cluster information. The target information includes at least: table structure information and authorization information. The authorization information includes at least: the authorized object to which the permission belongs and the permission name.

[0031] For example, to synchronize data information from Hive table a in source cluster A to Hive table b in target cluster B, at least the table structure information and authorization information (i.e., the authorization information in this application) of the Hive table a to be synchronized must also be synchronized to the target cluster B.

[0032] Specifically, before synchronizing the data in Hive table a to Hive table b in the target cluster B, it is necessary to query the data table information of the target cluster B based on the name of Hive table a.

[0033] Optionally, in the method for synchronizing Hive data tables between clusters provided in the embodiments of this application, before synchronizing the processed target information to the target Hive data table in the target cluster's Hive database, the method further includes: in response to a query command input on the target interface, obtaining the database name of the target cluster's Hive database and the table name of the target Hive data table, wherein the query command is used to query information in the target cluster.

[0034] Specifically, the target interface of the third-party platform graphically prompts the user to input: the name of the source cluster to be synchronized, the name of the target cluster to be synchronized, the Hive database to which the table to be synchronized belongs, and the name of the Hive table to be synchronized. After the user completes the input and selects execution, the system can not only extract the CLUSTER_ID of the source and target clusters in the background, but also obtain the DATABASE name of the Hive database to be synchronized and the TABLE_NAME name of the Hive table to be synchronized in the target cluster. By querying the Hive table in the source cluster corresponding to the Hive table in the target cluster to be synchronized, the accuracy of subsequent synchronization of the data tables to be synchronized is improved.

[0035] After obtaining the target cluster through the above steps, it is necessary to acquire the specific table structure information of the Hive data table to be synchronized. Optionally, in the method for synchronizing Hive data tables between clusters provided in this application embodiment, determining one of the target information of the Hive data table to be synchronized based on the source cluster information includes: acquiring the Hive interface information in the source cluster; connecting to the Hive database in the source cluster based on the Hive interface information; responding to the output instruction for table structure information, outputting the table structure information of the Hive database in the source cluster to obtain the output table structure information; and using the output table structure information as one of the target information.

[0036] For example, using the Hive interface to connect to the source cluster's Hive library, the Hive table structure statement to be synchronized can be exported. This Hive table structure statement can then be used to create a new table. Specifically, the table creation statement can be obtained by executing the following command in beeline: `beeline --entirelineascommand=false -e "set role admin; show create table[DATABASE].[TABLE_NAME];">>insert.hql`. This application improves the synchronization efficiency of data tables between different clusters by automatically obtaining table structure information.

[0037] After obtaining the specific table structure information of the Hive data table to be synchronized through the above steps, it is necessary to obtain the specific table authorization information of the Hive data table to be synchronized. Optionally, in the method for synchronizing Hive data tables between clusters provided in this application embodiment, determining one of the target information of the Hive data table to be synchronized based on the source cluster information includes: obtaining Hive metadata in the source cluster; determining multiple metadata tables in the Hive metadata; extracting the database ID information of Hive in the source cluster from the multiple metadata tables; and determining one of the target information based on the database ID information of Hive in the source cluster.

[0038] Specifically, determining one of the target information based on the Hive database ID information in the source cluster includes: extracting the ID information of the Hive data table to be synchronized based on the name of the Hive data table to be synchronized and the ID information of the Hive database in the source cluster; extracting the authorization information of the Hive data table to be synchronized from multiple authorization information tables based on the ID information of the Hive data table to be synchronized; and using the authorization information of the Hive data table to be synchronized as one of the target information.

[0039] For example, the application connects to the Hive metadata of the source cluster (such as Hivemeta of DBService), extracts the DB_ID (the ID information of the Hive database in the source cluster) from multiple metadata tables, and extracts the TBL_ID (the ID information of the Hive data table to be synchronized) from multiple metadata tables based on the DB_ID and TABLE_NAME (the name of the Hive data table to be synchronized). Based on the TBL_ID, it extracts all the authorization information of the source table from the TBL_PRIVS table (multiple authorization information tables). The authorization information includes the authorization information table (TBL_PRIV), the user role type to which the permission belongs (PRINCIPAL_TYPE), the authorization name (PRINCIPAL_NAME), and the granted permission (GRANT_OPTION). This application improves the synchronization accuracy of data tables between different clusters by obtaining the authorization information of the Hive data table to be synchronized.

[0040] Step S103: Process the target information to obtain the processed target information.

[0041] Step S104: Synchronize the processed target information to the target Hive data table in the Hive database of the target cluster.

[0042] Specifically, the target cluster is modified using the user role ID corresponding to that in the source cluster to synchronize Hive tables and table authorizations across different Hadoop clusters. The specific operations are as follows:

[0043] Optionally, in the method for synchronizing Hive data tables between clusters provided in this application embodiment, processing the target information to obtain the processed target information includes: creating a target data table structure based on table structure information, wherein the target data table structure is used to store the Hive data table to be synchronized; when the authorized object to which the permission belongs in the authorization information is a user group, obtaining the group role name in the user group, determining the role ID information of the same group in the target cluster based on the group role name, determining the target authorization name corresponding to the user group based on the role ID information, replacing the authorization name in the source cluster with the target authorization name to obtain the authorization information required by the target cluster; and using the target data table structure and the authorization information required by the target cluster as the processed target information.

[0044] For example, when the user role type to which the permission belongs is "user": the entry does not need further processing. When the user role type to which the permission belongs is "user group", the last 14 characters of the user role type field (i.e., underscores plus a 13-digit timestamp) are extracted to obtain the group role name in the user group. Based on the group role name in the user group, the target cluster metadata is queried to obtain the user role ID information with the same name in the target cluster. This ID information is used as the permission name required for authorization in the target cluster. The permission name required for authorization in the target cluster is overwritten with the permission name in the source cluster, that is, the last 14 characters of the user role type field to which the permission belongs in the source cluster Hive data table (i.e., underscores plus a 13-digit timestamp) are replaced. By modifying the authorization information when synchronizing different clusters, the synchronization efficiency of the Hive data table is improved.

[0045] Optionally, after processing the authorization information as described above, this application can generate an authorization statement from the processed authorization information (authorization information). For example, when GRANT_OPTION is 0: the target user is not allowed to sub-authorize the authorization information; when GRANT_OPTION is 1: the target user is allowed to sub-authorize the authorization information.

[0046] Optionally, in the method for synchronizing Hive data tables between clusters provided in this application embodiment, synchronizing the processed target information to the target Hive data table in the target cluster's Hive database includes: obtaining target Hive interface information in the target cluster; connecting to the target cluster's Hive database according to the target Hive interface information; and synchronizing the processed target information to the target Hive data table in the target cluster's Hive database in response to a synchronization command.

[0047] Specifically, by connecting to the target cluster's Hive library through the target Hive interface (such as beeline), executing inster.hql and the corresponding grant statement completes the table creation and authorization operations, which is the synchronization operation. This application can greatly simplify the implementation of table-level synchronization operations by the cluster operation and maintenance department through the synchronization of Hive tables with the same content across clusters, and also improves the convenience of subsequent testing work to a certain extent.

[0048] In summary, the method for synchronizing Hive data tables between clusters provided in this application involves: obtaining source cluster information, which includes at least the name of the source cluster, the name of the Hive database in the source cluster, and the name of the Hive data table to be synchronized in the Hive database of the source cluster; determining the target information of the Hive data table to be synchronized based on the source cluster information, where the target information includes at least table structure information and authorization information, where the authorization information includes at least the authorized object to which the permission belongs and the permission name; processing the target information to obtain processed target information; and synchronizing the processed target information to the target Hive data table in the Hive database of the target cluster. This application solves the problem of low synchronization efficiency of Hive tables across clusters in related technologies. By synchronizing the processed target information to the target Hive data table in the Hive database of the target cluster, the synchronization efficiency of Hive tables across clusters is improved.

[0049] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0050] This application also provides a device for synchronizing Hive data tables between clusters. It should be noted that this device can be used to execute the method for synchronizing Hive data tables between clusters provided in this application. The following describes the device for synchronizing Hive data tables between clusters provided in this application.

[0051] Figure 2 This is a schematic diagram of a device for synchronizing Hive data tables between clusters according to an embodiment of this application. Figure 2 As shown, the device includes: an acquisition unit 201, a determination unit 202, a processing unit 203, and a synchronization unit 204.

[0052] Specifically, the acquisition unit 201 is used to acquire source cluster information, wherein the source cluster information includes at least: the name of the source cluster, the name of the Hive database in the source cluster, and the name of the Hive data table to be synchronized in the Hive database of the source cluster;

[0053] The determining unit 202 is used to determine the target information of the Hive data table to be synchronized based on the source cluster information. The target information includes at least: table structure information and authorization information. The authorization information includes at least: the authorization object to which the permission belongs and the permission name.

[0054] The processing unit 203 is used to process the target information to obtain the processed target information;

[0055] Synchronization unit 204 is used to synchronize the processed target information to the target Hive data table in the Hive database of the target cluster.

[0056] In summary, the Hive data table synchronization device between clusters provided in this application embodiment obtains source cluster information through the acquisition unit 201, wherein the source cluster information includes at least: the name of the source cluster, the name of the Hive database in the source cluster, and the name of the Hive data table to be synchronized in the Hive database in the source cluster; the determination unit 202 determines the target information of the Hive data table to be synchronized based on the source cluster information, wherein the target information includes at least: table structure information and authorization information, wherein the authorization information includes at least: the authorized object to which the permission belongs and the permission name; the processing unit 203 processes the target information to obtain the processed target information; and the synchronization unit 204 synchronizes the processed target information to the target Hive data table in the Hive database of the target cluster, thereby solving the problem of low synchronization efficiency of Hive tables across clusters in related technologies. By synchronizing the processed target information to the target Hive data table in the Hive database of the target cluster, the synchronization efficiency of Hive tables across clusters is improved.

[0057] Optionally, in the Hive data table synchronization device between clusters provided in the embodiments of this application, the device further includes: a response unit, used to obtain the database name of the Hive database of the target cluster and the table name of the target Hive data table in response to a query command input on the target interface before synchronizing the processed target information to the target Hive data table in the Hive database of the target cluster, wherein the query command is used to query the information in the target cluster.

[0058] Optionally, in the Hive data table synchronization device between clusters provided in this application embodiment, the determining unit includes: a first acquisition module, used to acquire Hive interface information in the source cluster; a first connection module, used to connect to the Hive database in the source cluster according to the Hive interface information; a first response module, used to output the table structure information of the Hive database in the source cluster in response to the output instruction for table structure information, to obtain the output table structure information; and a first determining module, used to use the output table structure information as one of the target information.

[0059] Optionally, in the Hive data table synchronization device between clusters provided in the embodiments of this application, the determining unit includes: a second acquisition module, used to acquire Hive metadata in the source cluster; a second determining module, used to determine multiple metadata tables in the Hive metadata; an extraction module, used to extract the database ID information of Hive in the source cluster from the multiple metadata tables; and a third determining module, used to determine one of the target information based on the database ID information of Hive in the source cluster.

[0060] Optionally, in the Hive data table synchronization device between clusters provided in this application embodiment, the third determining module includes: a first extraction submodule, used to extract the ID information of the Hive data table to be synchronized based on the name of the Hive data table to be synchronized and the ID information of the Hive database in the source cluster; a second extraction submodule, used to extract the authorization information of the Hive data table to be synchronized from multiple authorization information tables based on the ID information of the Hive data table to be synchronized; and a determining submodule, used to use the authorization information of the Hive data table to be synchronized as one of the target information.

[0061] Optionally, in the Hive data table synchronization device between clusters provided in this application embodiment, the processing unit includes: a creation module, used to create a target data table structure according to table structure information, wherein the target data table structure is used to store the Hive data table to be synchronized; a fourth determination module, used to obtain the group role name in the user group when the authorized object to which the permission belongs in the authorization information is a user group, determine the role ID information of the same group in the target cluster according to the group role name, determine the target authorization name corresponding to the user group according to the role ID information, replace the authorization name in the source cluster according to the target authorization name, and obtain the authorization information required by the target cluster; and a fifth determination module, used to use the target data table structure and the authorization information required by the target cluster as the processed target information.

[0062] Optionally, in the Hive data table synchronization device between clusters provided in the embodiments of this application, the synchronization unit includes: a third acquisition module, used to acquire target Hive interface information in the target cluster; a second connection module, used to connect to the Hive database of the target cluster according to the target Hive interface information; and a second response module, used to synchronize the processed target information to the target Hive data table in the Hive database of the target cluster in response to the synchronization command.

[0063] The synchronization device for Hive data tables between clusters includes a processor and a memory. The aforementioned acquisition unit 201, determination unit 202, processing unit 203, synchronization unit 204, etc., are all stored in the memory as program units. The processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0064] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and kernel parameters are adjusted to synchronize Hive data tables across the cluster.

[0065] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0066] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements a method for synchronizing Hive data tables between clusters.

[0067] This invention provides a processor for running a program, wherein the program executes a method for synchronizing Hive data tables between clusters during runtime.

[0068] like Figure 3 As shown, this embodiment of the invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: obtaining source cluster information, wherein the source cluster information includes at least: the name of the source cluster, the name of the Hive database in the source cluster, and the name of the Hive data table to be synchronized in the Hive database in the source cluster; determining the target information of the Hive data table to be synchronized based on the source cluster information, wherein the target information includes at least: table structure information and authorization information, wherein the authorization information includes at least: the authorized object to which the permission belongs and the permission name; processing the target information to obtain processed target information; and synchronizing the processed target information to the target Hive data table in the Hive database of the target cluster.

[0069] When the processor executes the program, it also performs the following steps: before synchronizing the processed target information to the target Hive data table in the target cluster's Hive database, in response to the query command entered on the target interface, it obtains the database name of the target cluster's Hive database and the table name of the target Hive data table, wherein the query command is used to query information in the target cluster.

[0070] When the processor executes the program, it also performs the following steps: obtains Hive interface information from the source cluster; connects to the Hive database in the source cluster based on the Hive interface information; responds to the instruction to output table structure information, outputs the table structure information of the Hive database in the source cluster, and obtains the output table structure information; and uses the output table structure information as one of the target information.

[0071] When the processor executes the program, it also performs the following steps: obtains Hive metadata from the source cluster; identifies multiple metadata tables in the Hive metadata; extracts the database ID information of Hive in the source cluster from the multiple metadata tables; and determines one of the target information based on the database ID information of Hive in the source cluster.

[0072] When the processor executes the program, it also performs the following steps: extracting the ID information of the Hive data table to be synchronized based on the name of the Hive data table to be synchronized and the ID information of the Hive database in the source cluster; extracting the authorization information of the Hive data table to be synchronized from multiple authorization information tables based on the ID information of the Hive data table to be synchronized; and using the authorization information of the Hive data table to be synchronized as one of the target information.

[0073] When the processor executes the program, it also performs the following steps: creating a target data table structure based on the table structure information, wherein the target data table structure is used to store the Hive data table to be synchronized; when the authorized object to which the permission belongs in the authorization information is a user group, obtaining the group role name in the user group, determining the role ID information of the group with the same name in the target cluster based on the group role name, determining the target authorization name corresponding to the user group based on the role ID information, replacing the authorization name in the source cluster with the target authorization name to obtain the authorization information required by the target cluster; and using the target data table structure and the authorization information required by the target cluster as the processed target information.

[0074] When the processor executes the program, it also performs the following steps: obtains the target Hive interface information in the target cluster; connects to the Hive database of the target cluster based on the target Hive interface information; and, in response to the synchronization command, synchronizes the processed target information to the target Hive data table in the Hive database of the target cluster.

[0075] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0076] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program comprising the following method steps: obtaining source cluster information, wherein the source cluster information includes at least: the name of the source cluster, the name of the Hive database in the source cluster, and the name of the Hive data table to be synchronized in the Hive database in the source cluster; determining the target information of the Hive data table to be synchronized based on the source cluster information, wherein the target information includes at least: table structure information and authorization information, wherein the authorization information includes at least: the authorization object to which the permission belongs and the permission name; processing the target information to obtain processed target information; and synchronizing the processed target information to the target Hive data table in the Hive database of the target cluster.

[0077] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: before synchronizing the processed target information to the target Hive data table in the target cluster's Hive database, in response to a query command entered on the target interface, obtain the database name of the target cluster's Hive database and the table name of the target Hive data table, wherein the query command is used to query information in the target cluster.

[0078] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: obtaining Hive interface information from the source cluster; connecting to the Hive database in the source cluster based on the Hive interface information; responding to the output instruction for table structure information, outputting the table structure information of the Hive database in the source cluster to obtain the output table structure information; and using the output table structure information as one of the target information.

[0079] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: obtaining Hive metadata from the source cluster; identifying multiple metadata tables in the Hive metadata; extracting the database ID information of Hive in the source cluster from the multiple metadata tables; and determining one of the target information based on the database ID information of Hive in the source cluster.

[0080] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: extracting the ID information of the Hive data table to be synchronized based on the name of the Hive data table to be synchronized and the ID information of the Hive database in the source cluster; extracting the authorization information of the Hive data table to be synchronized from multiple authorization information tables based on the ID information of the Hive data table to be synchronized; and using the authorization information of the Hive data table to be synchronized as one of the target information.

[0081] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: creating a target data table structure based on table structure information, wherein the target data table structure is used to store the Hive data table to be synchronized; when the authorized object to which the permission belongs in the authorization information is a user group, obtaining the group role name in the user group, determining the role ID information of the same group in the target cluster based on the group role name, determining the target authorization name corresponding to the user group based on the role ID information, replacing the authorization name in the source cluster with the target authorization name to obtain the authorization information required by the target cluster; and using the target data table structure and the authorization information required by the target cluster as the processed target information.

[0082] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: obtaining target Hive interface information in the target cluster; connecting to the Hive database of the target cluster based on the target Hive interface information; and synchronizing the processed target information to the target Hive data table in the Hive database of the target cluster in response to a synchronization command.

[0083] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0084] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0085] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0086] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0087] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0088] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0089] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0090] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0091] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for synchronizing Hive data tables between clusters, characterized in that, include: Obtain source cluster information, wherein the source cluster information includes at least: the name of the source cluster, the name of the Hive database in the source cluster, and the name of the Hive data table to be synchronized in the Hive database in the source cluster; The target information of the Hive data table to be synchronized is determined based on the source cluster information, wherein the target information includes at least: table structure information and authorization information, wherein the authorization information includes at least: the authorization object to which the permission belongs and the permission name; The target information is processed to obtain processed target information; The processed target information is synchronized to the target Hive data table in the Hive database of the target cluster; The processed target information obtained by processing the target information includes: A target data table structure is created based on the table structure information, wherein the target data table structure is used to store the Hive data table to be synchronized; When the authorized object of the permission in the authorization information is a user group, the group role name in the user group is obtained by extracting a preset length field after the user role type field of the permission. Based on the group role name, the target cluster metadata is queried to determine the role ID information of groups with the same name in the target cluster. Based on the role ID information, the target authorization name corresponding to the user group is determined. The authorization name after the preset length field of the user role type field of the permission in the Hive data table of the source cluster is replaced with the target authorization name to obtain the authorization information required by the target cluster; and... The target data table structure and the authorization information required by the target cluster are used as the processed target information; The method further includes: after processing the authorization information, generating an authorization statement based on the processed authorization information, wherein the authorization statement contains control information for indicating whether the target user is allowed to sub-authorize the authorization information; The method further includes: connecting to the Hive metadata of the source cluster and extracting the ID information of the Hive database from multiple metadata tables; extracting the ID information of the Hive data table to be synchronized from multiple metadata tables based on the ID information of the Hive database and the name of the Hive data table to be synchronized in the Hive database of the source cluster; and extracting all authorization information of the Hive data table to be synchronized from multiple authorization information tables based on the ID information of the Hive data table to be synchronized.

2. The method of claim 1, wherein, Before synchronizing the processed target information to the target Hive data table in the Hive database of the target cluster, the method further includes: In response to a query command entered on the target interface, the database name of the Hive database of the target cluster and the table name of the target Hive data table are obtained, wherein the query command is used to query information in the target cluster.

3. The method of claim 1, wherein, Based on the source cluster information, one of the target information for the Hive data table to be synchronized includes: Obtain the Hive interface information from the source cluster; Connect to the Hive database in the source cluster based on the Hive interface information; In response to the output instruction for the table structure information, the table structure information of the Hive database in the source cluster is output to obtain the output table structure information; The output table structure information is used as one of the target information.

4. The method of claim 1, wherein, Based on the source cluster information, one of the target information for the Hive data table to be synchronized includes: Obtain Hive metadata from the source cluster; Identify multiple metadata tables within the Hive metadata; Extract the database ID information of Hive in the source cluster from the multiple metadata tables; One of the target information is determined based on the ID information of the Hive database in the source cluster.

5. The method of claim 4, wherein, Determining one of the target information based on the ID information of the Hive database in the source cluster includes: Extract the ID information of the Hive data table to be synchronized based on the name of the Hive data table to be synchronized and the ID information of the Hive database in the source cluster; Based on the ID information of the Hive data table to be synchronized, the authorization information of the Hive data table to be synchronized is extracted from multiple authorization information tables; The authorization information of the Hive data table to be synchronized is used as one of the target information.

6. The method of claim 1, wherein, The target Hive data table that synchronizes the processed target information to the Hive database of the target cluster includes: Obtain the target Hive interface information in the target cluster; Connect to the Hive database of the target cluster according to the target Hive interface information; In response to the synchronization command, the processed target information is synchronized to the target Hive data table in the Hive database of the target cluster.

7. A device for synchronizing Hive data tables between clusters, characterized in that, include: The acquisition unit is used to acquire source cluster information, wherein the source cluster information includes at least: the name of the source cluster, the name of the Hive database in the source cluster, and the name of the Hive data table to be synchronized in the Hive database in the source cluster; The determining unit is configured to determine the target information of the Hive data table to be synchronized based on the source cluster information, wherein the target information includes at least: table structure information and authorization information, wherein the authorization information includes at least: the authorization object to which the permission belongs and the permission name; The processing unit is used to process the target information to obtain processed target information; The synchronization unit is used to synchronize the processed target information to the target Hive data table in the Hive database of the target cluster; The processing unit includes: a creation module, used to create a target data table structure based on table structure information, wherein the target data table structure is used to store the Hive data table to be synchronized; a fourth determination module, used to, when the authorized object to which the permission belongs in the authorization information is a user group, obtain the group role name in the user group by extracting the user role type field to which the permission belongs and a preset length field, query the target cluster metadata based on the group role name to determine the role ID information of the same group in the target cluster, determine the target authorization name corresponding to the user group based on the role ID information, and replace the authorization name in the preset length field after the user role type field to which the permission belongs in the Hive data table of the source cluster with the target authorization name to obtain the authorization information required by the target cluster; and a fifth determination module, used to use the target data table structure and the authorization information required by the target cluster as the processed target information; The device further includes: after processing the authorization information, generating an authorization statement based on the processed authorization information, wherein the authorization statement contains control information for indicating whether the target user is allowed to sub-authorize the authorization information; The device is also used to connect to the Hive metadata of the source cluster, extract the ID information of the Hive database from multiple metadata tables; extract the ID information of the Hive data table to be synchronized from multiple metadata tables based on the ID information of the Hive database and the name of the Hive data table to be synchronized in the Hive database of the source cluster; and extract all authorization information of the Hive data table to be synchronized from multiple authorization information tables based on the ID information of the Hive data table to be synchronized.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program, wherein the program executes the method described in any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 6.

Images