Data warehouse access control method, system and electronic device

Abstract

The present invention provides a data warehouse access control method, system and computer equipment, the method comprising: storing authorized account association information, and updating the authorized account association information by periodically acquiring metadata, the authorized account association information including authorized account information and its corresponding database information; establishing an account authentication service independent of the database access service and metadata access service in the data warehouse to call the stored authorized account association information; responding to the access request of the client; calling the account authentication service to identify and process the account information in the access request according to the stored authorized account association information to perform identity authentication; returning the user request that passes the identity authentication to the database access service and metadata access service for permission verification. The present invention reliably guarantees the security of data in the Hive data warehouse, realizes a more effective identity authentication and permission verification process, and effectively prevents user misoperation and malicious operation.

Description

Technical Field

[0001] The present invention relates to the field of testing technology, and in particular to a data warehouse access control method, system and electronic equipment. Background Art

[0002] Internet-based applications often require the exchange of resources between different parties. Resources here refer to any available material, information, money, time, and so on. Information resources include computing resources and various types of data resources. Data resources include specialized data in various fields.

[0003] Data is a critical asset for enterprises, and data security and controllable access are crucial. Hive, as the gateway to big data storage and access, has become a focus of industry attention. Permission management frameworks such as Ranger and Sentry have emerged. However, these frameworks focus on centralized management of the HiveServer2 server and cannot authenticate accounts or verify permissions on the metadata server.

[0004] Currently, HiveServer2 supports multiple user authentication methods: NONE, NOSASL, KERBEROS, LDAP, PAM, and CUSTOM. Kerberos, LDAP, PAM, and CUSTOM are considered security management authentication methods. However, these authentication methods all have different drawbacks in practical applications. Kerberos and LDAP use login usernames for development machines. This authentication method results in a large number of public accounts being unavailable to personal accounts, resulting in a significant legacy burden and difficult migration. When using PAM, if a user's password has expired, the JPAM library used to provide the PAM authentication mode may cause HiveServer2 to shut down. In other cases, some users may experience crashes during login. In actual production environments, each account addition requires restarting the Hive system to load the configuration file, making it unsuitable for company-wide, large-scale user account additions. This can lead to unreasonable or inappropriate user account management. Furthermore, the Hive system does not support account and password verification methods stored in databases (such as MySQL), and the Hive system cannot implement permission control for access requests directly to the metadata repository. Furthermore, there are technical issues such as an inability to effectively prevent user errors and malicious operations, resulting in low data security.

[0005] Therefore, it is necessary to provide a more effective access control method for data warehouses. Summary of the Invention

[0006] The existing solution has the following problems: the Hive system cannot support the account and password verification method stored in MYSQL, Hive cannot perform permission control on access requests directly connected to the metadata database, and has low data security and other technical issues.

[0007] In view of the above problems, the first aspect of the present invention provides an access control method for a data warehouse, comprising: storing authorized account association information, and updating the authorized account association information by periodically obtaining metadata, wherein the authorized account association information includes authorized account number information and its corresponding database information; establishing an account authentication service independent of the database access service and the metadata access service in the data warehouse to call the stored authorized account association information; responding to an access request from a client; calling the account authentication service to identify and process the account information in the access request based on the stored authorized account association information to perform identity authentication; and returning the user request that has passed the identity authentication to the database access service and the metadata access service for permission verification.

[0008] According to an optional embodiment of the present invention, when the access request is a metadata access request, the account information in the access request is passed to the metadata access service through the user group information, and the account authentication service is called to perform identity authentication.

[0009] According to an optional implementation manner of the present invention, delivering the account information in the access request to the metadata access service via the user group information includes: adding a specific identifier to a prefix of the user group information.

[0010] According to an optional embodiment of the present invention, the permission verification includes: when the access request is a metadata access request, monitoring an execution event of the metadata access request through a hook function, and performing permission verification when the execution event is performed.

[0011] According to an optional implementation manner of the present invention, the access request is an access request to the Metastore of the Hive data warehouse, and the permission verification includes: adding permission verification for database tables and databases to the permission verification of the original Metastore.

[0012] According to an optional implementation manner of the present invention, it also includes: setting an access transition period for the data warehouse and an account whitelist corresponding to the access transition period; during the access transition period, when the access request does not contain an account name and password, determining whether the account information in the access request is in the whitelist; if so, treating the access request as an access request for a public account, and not performing identity authentication and permission management on the access request for the public account.

[0013] According to an optional implementation manner of the present invention, the method further includes: regularly retrieving source information corresponding to a public account from an audit log of a data warehouse, and sending a prompt message to the public account based on the source information.

[0014] According to an optional implementation manner of the present invention, at the end of the transition period, the whitelist is deleted, and identity authentication and rights management are performed on all access requests to the database access service and the metadata access service.

[0015] In addition, the second aspect of the present invention provides an access control system for a data warehouse, comprising: a storage module for storing authorized account association information and updating the authorized account association information by periodically obtaining metadata, wherein the authorized account association information includes authorized account information and its corresponding database information; an establishment module for establishing an account authentication service independent of the database access service and the metadata access service within the data warehouse to call the stored authorized account association information; a receiving module for responding to an access request from a client; a calling module for calling the account authentication service to identify and process the account information in the access request based on the stored authorized account association information to perform identity authentication; and a verification module for returning the user request that has passed the identity authentication to the database access service and the metadata access service for permission verification.

[0016] According to an optional embodiment of the present invention, when the access request is a metadata access request, the account information in the access request is passed to the metadata access service through the user group information, and the account authentication service is called to perform identity authentication.

[0017] According to an optional implementation manner of the present invention, the transmitting the account name and password in the access request to the metadata access service via the user group information includes: adding a specific identifier to a prefix of the user group information.

[0018] According to an optional embodiment of the present invention, the permission verification includes: when the access request is a metadata access request, monitoring an execution event of the metadata access request through a hook function, and performing permission verification when the execution event is performed.

[0019] According to an optional implementation manner of the present invention, the access request is an access request to the Metastore of the Hive data warehouse, and the permission verification includes: adding permission verification for database tables and databases to the permission verification of the original Metastore.

[0020] According to an optional embodiment of the present invention, a setting module is further included, which is used to set an access transition period for the data warehouse and an account whitelist corresponding to the access transition period; during the access transition period, when the access request does not contain an account name and password, it is determined whether the account information in the access request is in the whitelist. If so, the access request is treated as an access request for a public account, and identity authentication and permission management are not performed on the access request for the public account.

[0021] According to an optional implementation manner of the present invention, the method further includes: regularly retrieving source information corresponding to a public account from an audit log of a data warehouse, and sending a prompt message to the public account based on the source information.

[0022] According to an optional implementation manner of the present invention, at the end of the transition period, the whitelist is deleted, and identity authentication and rights management are performed on all access requests to the database access service and the metadata access service.

[0023] In addition, the third aspect of the present invention provides a computer device, wherein the computer device includes: a processor; and a memory storing computer-executable instructions, wherein the executable instructions, when executed, enable the processor to execute the data warehouse access control method described in the first aspect of the present invention.

[0024] In addition, the fourth aspect of the present invention provides a computer program product, wherein the computer program product stores one or more programs, and when the one or more programs are executed by a processor, they implement the data warehouse access control method described in the first aspect of the present invention.

[0025] Beneficial effects

[0026] Compared with the existing technology, the present invention can provide more accurate data basis for account authentication service by storing authorized account association information and periodically updating the authorized account association information; establish an effective account authentication service, and call the stored authorized account association information to perform account authentication service and authenticate access requests, which can reliably ensure the security of data in the Hive data warehouse, realize more effective identity authentication and permission verification process, and effectively prevent user misoperation and malicious operation.

[0027] Furthermore, by taking the approach of periodic data updates, the problem of online account authentication failure caused by inconsistency between the authorized account association information in the account authentication module and the database information in the metadata database can be effectively avoided; by adding a local database (or local MAP) to the metadata database, not only can it be ensured that when there is a problem with accessing the metadata database, the relevant data in the local database is used for authentication, but the access pressure on the metadata database can also be greatly reduced; by adding an access transition period and an account whitelist corresponding to the access transition period, and increasing the whitelist judgment step, it is possible to ensure normal access to the old access method during the access transition period, that is, the transition period is compatible with the old method, so as to effectively avoid affecting the online production environment; by calling the account authentication service, identity authentication is performed on access requests from different clients to the database server and the metadata server, which can limit malicious access by illegal users using non-upgraded clients and achieve a more effective identity authentication process; it can realize user group information for multiple data requests in the same connection, and can distinguish user group information of clients with different connections, thereby enabling more effective permission verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to make the technical problems solved by the present invention, the technical means adopted, and the technical effects achieved more clearly, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, it should be noted that the drawings described below are only drawings of exemplary embodiments of the present invention. Those skilled in the art can derive drawings of other embodiments based on these drawings without inventive effort.

[0029] Figure 1 This is a flowchart of an example of a data warehouse access control method according to the first embodiment of the present invention.

[0030] Figure 2 This is a schematic block diagram of an application scenario of the data warehouse access control method according to embodiment 1 of the present invention.

[0031] Figure 3 It is a schematic diagram of another example of the data warehouse access control method according to the first embodiment of the present invention.

[0032] Figure 4 It is a schematic block diagram of the access entry in the data warehouse access control method of embodiment 1 of the present invention.

[0033] Figure 5 It is a schematic block diagram of the account authentication module in the data warehouse access control method of Example 1 of the present invention.

[0034] Figure 6 This is a schematic diagram of an example of a data warehouse access control system according to the second embodiment of the present invention.

[0035] Figure 7 This is a schematic diagram of another example of the access control system for a data warehouse according to the second embodiment of the present invention.

[0036] Figure 8 is a structural block diagram of an exemplary embodiment of a computer device according to the present invention.

[0037] Figure 9 is a block diagram of an exemplary embodiment of a computer program product according to the present invention. DETAILED DESCRIPTION

[0038] Exemplary embodiments of the present invention will now be described more fully with reference to the accompanying drawings. However, exemplary embodiments can be implemented in various forms, and it should not be understood that the present invention is limited to the embodiments set forth herein. On the contrary, providing these exemplary embodiments enables the present invention to be more comprehensive and complete, making it easier to fully convey the inventive concept to those skilled in the art. In the figures, the same reference numerals represent the same or similar elements, components or parts, and thus their repeated description will be omitted.

[0039] Under the premise of being consistent with the technical concept of the present invention, the features, structures, characteristics or other details described in a specific embodiment do not exclude that they can be combined in one or more other embodiments in a suitable manner.

[0040] In the description of specific embodiments, the features, structures, characteristics, or other details of the present invention are described to enable those skilled in the art to fully understand the embodiments. However, this does not preclude those skilled in the art from practicing the technical solutions of the present invention without one or more of the specific features, structures, characteristics, or other details.

[0041] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0042] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0043] It should be understood that while the terms "first," "second," and "third" may be used herein to describe various devices, elements, components, or parts, this should not be construed as limiting. These terms are used to distinguish one from another. For example, a first device could also be referred to as a second device without departing from the essential technical solution of the present invention.

[0044] The term "and / or" or "and / or" includes any one and all combinations of one or more of the associated listed items.

[0045] In the prior art, the operations that users perform on data in data warehouses such as Hive are irreversible. Therefore, in order to prevent users from making mistakes and performing malicious operations, it is necessary to confirm the identity of users when accessing the data warehouse (authentication), and at the same time limit the operations that users can perform (authorization). Therefore, in order to further optimize the data warehouse authority management, research and development improvements are made from the two parts of account authentication and authority management, and the data warehouse access control method of the present invention is proposed. This method can provide more accurate data basis for account authentication services by storing authorized account association information and periodically updating the authorized account association information; establish an effective account authentication service, and call the stored authorized account association information to perform account authentication services, authenticate access requests, and reliably ensure the security of data in the data warehouse, and can achieve more effective identity authentication and authority verification processes, which can effectively prevent users from making mistakes and performing malicious operations.

[0046] It should be noted that in the present invention, the data warehouse is a subject-oriented, integrated, relatively stable (non-volatile), and time-variant data set. The data warehouse is subject-oriented, and the data in the data warehouse is organized according to certain subject domains. A subject refers to the key aspects that users are concerned about when using the data warehouse to make decisions. A subject is usually related to multiple operational information systems. The data warehouse is integrated, and its data comes from scattered operational data. The required data is extracted from the original data, processed and integrated, and can only be entered into the data warehouse after unification and integration. For example, a hive data warehouse based on Hadoop is used for data extraction, transformation, and loading. It is a mechanism that can store, query, and analyze large-scale data stored in Hadoop. It can map structured data files into a database table, provide SQL query functions, and convert SQL statements into MapReduce tasks for execution. In addition, the Hive data warehouse has the functions of data storage and processing.

[0047] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0048] Example 1

[0049] Below, we will refer to Figures 1 to 5 An embodiment of the data warehouse access control method of the present invention is described.

[0050] The access control method of the present invention is used for data warehouse access control and is particularly suitable for data warehouses that provide database access services and metadata access services, such as Hive-based data warehouses (hereinafter referred to as Hive data warehouses). Specifically, the data warehouse of the present invention provides database access services and metadata access services, and the metadata describes the database in the data warehouse. Specifically, the metadata includes the table name, the database to which the table belongs (default by default), the table's columns and partitions and their attributes, the table type (whether it is an external table, etc.), and the directory where the table's data resides.

[0051] Figure 1 FIG. 1 is a flow chart of the data warehouse access control method of the present invention. Figure 1 As shown, the method includes the following steps.

[0052] Step S101 : storing the authorized account association information and updating the authorized account association information by periodically acquiring metadata. The authorized account association information includes the authorized account number information and its corresponding database information.

[0053] Specifically, the data warehouse includes database access services and metadata access services.

[0054] In this step, the authorized account association information that can access the data warehouse is stored, and the latest metadata is periodically obtained from the metadata repository to update the corresponding authorization association information, wherein the metadata includes the table name, the database to which the table belongs (the default is default), the columns and partitions of the table and their attributes, the type of the table (whether it is an external table, etc.), the directory where the table data is located, etc.; the authorized account association information includes the authorized account information and its corresponding database information, the authorized account information includes the account ID, account name (or user name), password, etc., and the database information includes the data access permission information corresponding to the authorized account, the data table where the data is located, the database to which the data table belongs, and other information.

[0055] Authorized account association information can be stored as a data table or database in a data warehouse, and an access interface can be provided.

[0056] To address the technical issues that the existing Hive system cannot support the account and password verification method stored in the database (such as MYSQL), and the Hive system cannot perform permission control on access requests directly connected to the metadata database, this solution establishes an account authentication service within the data warehouse.

[0057] Specifically, in step S102, an account authentication service independent of the database access service and the metadata access service is established in the data warehouse to call the stored authorized account association information.

[0058] More specifically, as a specific implementation method, an authentication module dedicated to user authentication can be configured for the data warehouse, and the authentication module includes a data table or database for storing account association information. In addition, the authentication module can periodically access the database for storing metadata in the data warehouse to pull the latest metadata, and use the latest metadata to update the authorized account association information. Thus, the authentication module can be called to authenticate the identity of users accessing the data warehouse, that is, the account authentication service is provided through the authentication module, so as to realize the account and password verification stored in the database (such as MYSQL), and to authenticate the access request of the Hive system (especially the access request directly connected to the metadata database).

[0059] Step S103: respond to the access request from the client.

[0060] Figure 2 This is a schematic block diagram of an application scenario of the data warehouse access control method according to embodiment 1 of the present invention.

[0061] like Figure 2 As shown, when a user wants to access the data warehouse, an access request is sent to the data warehouse through a client, wherein the client includes a first type of client and a second type of client for accessing a database server and a metadata server, the database server is used to provide database access services, and the metadata server is used to provide metadata access services.

[0062] Users of data warehouses (especially Hive data warehouses) include R&D engineers, data analysts, product managers, operations personnel, and other related business personnel.

[0063] Step S104 : calling the account authentication service to identify the account information in the access request according to the stored authorized account association information to perform identity authentication.

[0064] In this step, upon receiving an access request from a client, the data warehouse invokes the account authentication service. In a specific embodiment, the authentication module is invoked to identify and process the account information (e.g., name and password) in the access request based on the authorized account association information stored by the authentication module to perform identity authentication. This enables a more reliable implementation of account and password verification methods stored in databases (e.g., MYSQL) and more efficient authentication of Hive system access requests (particularly those directly connected to the metadata repository).

[0065] To further address the technical issue of the Hive system being unable to manage permissions for access requests directly connected to the metadata database, this solution uses an account authentication module to further verify permissions.

[0066] Specifically, in step S105, the user request that passes the identity authentication is returned to the database access service and the metadata access service for authority verification.

[0067] Based on the identity authentication result of the authentication module, the authenticated user request is returned to the database access service or metadata access service for further permission verification. This allows for more efficient permission verification of all access requests to the Hive system (including those directly connected to the metadata database).

[0068] Furthermore, when the user request passes the authority verification, corresponding data is returned to the client.

[0069] For user requests that fail identity authentication, a response message such as identity authentication failure is returned to the client.

[0070] For user requests that fail the authority verification, a response message such as identity authentication failure is returned to the client.

[0071] It should be noted that the above description is only provided as an example and should not be construed as limiting the present invention.

[0072] Figure 3 It is a schematic diagram of another example of the data warehouse access control method according to the first embodiment of the present invention.

[0073] First, in step S201, the authorization account association information is stored and updated by periodically acquiring metadata. The authorization account association information includes authorization account information and its corresponding database information.

[0074] Figure 4 It is a schematic block diagram of the access entry in the data warehouse access control method of embodiment 1 of the present invention.

[0075] Specifically, the Hive data warehouse includes a database access service and a metadata access service, and includes a first access entry and a second access entry corresponding to the database access service and the metadata access service. Figure 4 .

[0076] Because different data has different confidentiality levels, permissions for different user behaviors (including the actions users can take on the data) are granted to the data in the Hive data warehouse based on the confidentiality level and permitted actions. For example, some data may be accessible to all users, while others may be accessible to only a limited number of users. Still other data may be viewed only by a select number of users, with some users allowed to download and save data, and others allowed to delete or add data. In other words, different accounts (i.e., different users) are granted different permissions, thus creating authorized accounts.

[0077] In order to implement the account and password verification method stored in the database (such as MYSQL) in the Hive system and to ensure the security of the data in the database, the Hive data warehouse of the present invention adds an authentication module, which is used to provide account authentication services to authenticate the client's access request, thereby more reliably ensuring data security.

[0078] Figure 5 It is a schematic block diagram of the account authentication module in the data warehouse access control method of Example 1 of the present invention.

[0079] like Figure 5 As shown, the account authentication module includes a local MAP, and uses the local MAP to store authorized account association information.

[0080] Specifically, the authorized account information includes the account number, account name, password, and database information corresponding to the account. The database information includes the data table where the data is located, the database to which the data table belongs, the database name, the data table name, the data table attributes, the location of the data table where the data is located, etc.

[0081] Therefore, by storing the authorized account association information and periodically updating the authorized account association information, a more accurate data basis can be provided for the account authentication service.

[0082] It should be noted that the above description is only provided as an example and should not be construed as limiting the present invention.

[0083] In order to realize identity authentication for multiple access methods and data access permission management, this solution establishes an account authentication service in the data warehouse, that is, provides account authentication service.

[0084] Next, in step S202, an account authentication service independent of the database access service and the metadata access service is established in the data warehouse. The account authentication service can call the stored authorization account association information.

[0085] In this example, the data warehouse also provides an account authentication service. Specifically, an account authentication module is used to provide an account authentication service that is independent of a database access service and a metadata access service.

[0086] Specifically, for example, based on Hive open source code, an account authentication module is developed by inheriting (for example, the org.apache.hive.service.auth.PasswdAuthenticationProvider class). Data source parameters are added to the configuration file to generate the module. For example, in an actual project, MYSQL can be used as the data source for storing account-related information. The data source parameters include the database address, port, database, account number, password, and data update period.

[0087] As a specific implementation, for the Hive data warehouse, the authentication module is implemented as a Jar package, which is initialized and loaded by a function internally. A separate global thread is also started to periodically access the database to pull the latest data. Periodic updates can prevent online account authentication failures due to database issues. In other words, the Jar package performs account authentication and returns True or False. The HiveServer and MetaServer sides of the Hive data warehouse provide database access and metadata access services, respectively. Both sides can authenticate users by referencing Jars and calling functions. This effectively enables user authentication in access requests to the HiveServer and MetaServer sides.

[0088] As a preferred embodiment, the latest data pulled from the database is periodically accessed and updated to the data structure MAP. Therefore, when a database access problem occurs, online account authentication can still use the data in the local MAP for authentication, and greatly reduces the access pressure on MYSQL as the authentication account information source.

[0089] It should be noted that the above description is only provided as an example and should not be construed as limiting the present invention.

[0090] Next, in step S203, an access request from the client is received.

[0091] Specifically, when a user wants to access the data warehouse, he sends an access request to the data warehouse through a client, wherein the client includes Figure 4The first access entry and the second access entry are shown. The first access entry corresponds to the database server that provides database services, and the second access entry corresponds to the metadata server that provides metadata services.

[0092] More specifically, there are two ways to access the Hive data warehouse: one is to access the Hive server (i.e., the database server), and the other is to directly access the Hive metadata server (i.e., the metadata server). For details, see Figure 2 You can access the Hive data warehouse in either of two ways to obtain the data corresponding to the access request.

[0093] It should be noted that the Hive data warehouse in this solution adds account authentication and permission authentication features. All services accessing the Hive data warehouse require account information to access. For example, when accessing the Hive data warehouse through clients such as Spark, Flink, Beeline, and Hive, account authentication (identity authentication) and permission management (or authorization verification) are required to ensure the security of the data in the Hive data warehouse.

[0094] Furthermore, according to the received access request, the account authentication module is used to perform account authentication, and response information is sent to the client, such as account authentication passed, account authentication passed, account abnormality and other response information.

[0095] Therefore, by authenticating the accounts of access requests from different clients, the security of data in the Hive data warehouse can be reliably guaranteed, and user misoperations and malicious operations can be effectively prevented.

[0096] In order to ensure compatibility with the original access method during the client upgrade process, this solution adds an access transition period and an account whitelist.

[0097] Specifically, in step S204, an access transition period of the data warehouse and an account whitelist corresponding to the access transition period are set.

[0098] In actual project use, functionality iteration is typically performed based on existing online clusters. For big data business departments, the average daily real-time and offline data access volume is extremely high, but in this case, it is also necessary to ensure that users are unaware and minimize the online impact.

[0099] For example, when there are upgrades to client or server components or software, in order to ensure smooth project upgrades while guaranteeing the security of data in the data warehouse, for different projects (such as educational service Paiso product projects), for example, based on project type, number of participants, data volume and other parameters, set a data warehouse access transition period and an account whitelist corresponding to the access transition period.

[0100] Specifically, a whitelist is configured according to the components or software upgraded by the client, and an access transition period corresponding to the upgrade process is set, or a time node (ie, Deadline) is set, wherein the access transition period is, for example, 1 to 3 months.

[0101] It should be noted that, in the present invention, the access transition period refers to the transition period of client upgrade (also called user upgrade), and the old access entrance will be closed after the end of the access transition period. The transition period before the end of the access transition period is the transition period of client upgrade. During the access transition period, the original access method, account authentication and permission verification method are compatible. In addition, the time node (Deadline) is a time node specified by the project service provider. Before this time node, the user is given a specific time period to upgrade the client to the latest version and verify whether the upgraded client is available. Before this time node, the original access method is still available to prevent affecting normal business. After the time node, the server will only allow access from the latest client. The Deadline time node and the client upgrade transition period are both intended to allow the entire project to be upgraded smoothly, giving users time to upgrade the client and debug the new client without affecting online production or the original access method.

[0102] For example, before account authentication, a whitelist check step is added. Through whitelist check, whitelist accounts are filtered before account authentication, thereby maintaining compatibility with the original (or old) access method of the original service account. This ensures normal access to the old access method during the access transition period, that is, compatibility with the old method during the transition period, effectively avoiding impact on the online production environment.

[0103] It should be noted that the access transition period and the whitelist are described above as examples only and should not be construed as limiting the present invention. In other examples, the access transition period may be increased or decreased depending on the project.

[0104] Specifically, during the access transition period, before performing account authentication, a determination is first made based on the whitelist whether the user corresponding to the access request is a user (or account) in the whitelist.

[0105] For example, when the access request does not include an account name and password, it is determined whether the account information in the access request is in the whitelist. If it is (that is, when it is determined that the user or account is in the whitelist), the access request is treated as an access request for a public account, and no identity authentication or permission management is performed on the access request for the public account.

[0106] For example, when the access request does not include an account name and password, the account passed by set_ugi is used to determine whether it is in the whitelist (ie, whether it is a public account).

[0107] Optionally, source information corresponding to the public account is retrieved from the audit log of the Hive data warehouse on a regular basis, and a prompt message is sent to the public account based on the source information.

[0108] For example, at hourly intervals, we extract public accounts from the Hive data warehouse's audit logs, process the data, and count the unmigrated traffic. We then identify all access requests from non-personal accounts as unmigrated traffic. We then identify the user or account based on the request source (IP address) and the table accessed. For example, we provide one-on-one upgrade guidance, answer questions in real time, and provide documentation introductions, thereby improving user upgrade efficiency.

[0109] When it is determined that the user or account is not in the whitelist (ie, if it is not), the account authentication module will be used to perform account authentication on the access request.

[0110] Furthermore, at the end of the access transition period, the whitelist is deleted, and identity authentication and permission management are performed on all access requests to the database access service and metadata access service. In other words, at the end of the access transition period, identity authentication and permission management are enabled for all access requests.

[0111] Therefore, by adding an access transition period and an account whitelist corresponding to the access transition period, and adding a whitelist judgment step, it is possible to ensure normal access to the old access method during the access transition period, that is, the transition period is compatible with the old method, so as to effectively avoid affecting the online production environment.

[0112] It should be noted that the above description is only provided as an example and should not be construed as limiting the present invention.

[0113] In order to provide account authentication services to a variety of different clients, an account authentication module is added. When an access request from a client is received, the account authentication service implemented by the account authentication module is called.

[0114] Specifically, in step S205, the account authentication service is called to identify the account information in the access request according to the stored authorized account association information to perform identity authentication (ie, account authentication).

[0115] Specifically, the account authentication module described in step S202 is used to call the account authentication service to perform identity authentication.

[0116] For example, when the access request is a metadata access request, the account name and password in the access request (for example, quhao and 12**6) are obtained, passed to the metadata access service through the user group information (for example, user1:passwd1), and the account authentication service is called for identity authentication.

[0117] Optionally, a specific identifier is added to the prefix of the user group information, such as @quhao:mima, where "@" is a specific identifier. In this embodiment, the user group information is @quhao:mima.

[0118] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention. In other examples, the specific identifier may also be represented by other symbols, such as “&”.

[0119] Specifically, when the client establishes a connection with the metadata server, the set_ugi method is used to provide the metadata server with user group information. However, this solution improves the set_ugi method by adding user group information (e.g., a string, such as quhao:mima; user:passwd; user:mima) and other data when storing information in set_ugi, and adding a specific identifier (e.g., @) to the prefix of the data to distinguish it from other content. This effectively authenticates the account information (including account number and password) in the access request, thereby achieving the function of transparently transmitting the account number and password. After obtaining the account number and password in the user group information, the metadata server uses the account authentication module to call the account authentication service (i.e., introduces the authentication function in the jar package such as hiveauth-1.0.jar), passes in the obtained account information (e.g., account number and password), determines whether the account number and password in the stored authorized account association information are consistent with the passed account number and password, transmits the authentication result to the metadata server, and then the metadata server returns a response message to the client indicating whether the identity authentication passed or failed. For example, the metadata access request is, for example, “hive-hiveconfhive.metastore.cu**om.se***ity.userpass=username:password”, and the user group information is, for example, “username=q**ao:password=mima”.

[0120] For example, when the access request is a database access request, user group information (specifically, including the account number (or account name) and password) is obtained, for example, through a session (i.e., memory temporarily created by the server). The account authentication module is used to call the account authentication service to determine whether the account number and password in the stored authorized account association information are consistent with the account information (e.g., account number or account name and password) in the access request. This is done to perform account authentication and transmit the authentication result to the database server. The database server then returns a response message indicating whether the authentication passed or failed to the client, such as a True or False response message. Specifically, the database access request may be, for example, "beeline -u "j dbc:hive2: / / ……%##@" -nusername -ppassword." The user group information is rewritten to, for example, user2:passwd2.

[0121] For example, the case where identity authentication fails also includes the case where the account and password are incorrect. In this case, a prompt message indicating the incorrect account or password is returned to the client.

[0122] It should be noted that the present invention can provide account authentication services for a variety of different clients. For account authentication services for different clients. For example, for the Hive client (i.e., Hiveclient), the hive-metastore.jar package can be directly replaced, for example, directly through the set command setting to perform account authentication services. For example, for the Spark client (i.e., Sparkclient), the hive-metastorejar package can be directly called, wherein the parameters need to be set through, for example, "-conf", and the spark.hadoop. prefix is ​​added before the parameter name to perform account authentication services.

[0123] Therefore, by calling the account authentication service to authenticate the access requests of different clients to the database server and metadata server, malicious access by illegal users using non-upgraded clients can be restricted, and a more efficient authentication process can be achieved.

[0124] It should be noted that the above description is only provided as an example and should not be construed as limiting the present invention.

[0125] In order to further solve the technical problem that the Hive system cannot perform permission control on access requests directly connected to the metadata database, this solution further performs permission verification by using the authentication module.

[0126] Specifically, in step S206, the user request that passes the identity authentication is returned to the database access service and the metadata access service for permission verification.

[0127] For example, when an access request to a database passes identity authentication, the access request is returned to the database access service for permission verification.

[0128] Specifically, the database server parses the data access information in the access request and splits the data access information into multiple execution events (for example, splitting it into accessing data table 1, which belongs to database k1, etc.). The database server determines whether the user has access rights based on the multiple execution events. During the access right verification, the permission verification for the database table and database is added to the permission verification of the original Metastore.

[0129] For example, when an access request to access metadata passes identity authentication (ie, when the access request is a metadata access request), the execution event of the metadata access request is monitored through a hook function, and permission verification is performed when the execution event is performed.

[0130] Specifically, for example, based on the authentication principle of Sentry itself, the permissions of the metadata database are verified. Specifically, in the authentication process of Sentry, by adding a hook function to the Hive data warehouse, the added hook function is called before and after the execution of the Hive data warehouse, such as before or after compilation, before or after execution. For example, the corresponding hook function can be called before or after the execution of the client's request command. For example, by adding a class such as "org.apache.sentry.binding.metastore.MetastoreAuthzBinding", this class inherits MetaStorePreEventListener and is called before the client's request command. In this way, permission verification (or permission control) is implemented, such as adding restrictions on gettable and get database. Therefore, while restricting the permissions for accessing the database server, the permissions for accessing the metadata server can also be restricted, thereby effectively implementing the permission restrictions for each access entry.

[0131] In another example, after the client calls set_ugi, the metadata server also needs to perform corresponding processing after receiving the set_ugi request, such as storing user group information in isolated memory. For example, a separate thread is assigned to each client connection using ThreadLocal variables. All ThreadLocal variables are used to bind the thread, thereby independently storing the user group information of each client and solving the problem of multi-concurrency.

[0132] It should be noted that the above threads are all placed in the thread pool, and the ThreadLocal variable needs to be cleared when the client connection is released.

[0133] In this way, user group information of multiple data requests in the same connection can be realized, and user group information of clients of different connections can be distinguished, thereby enabling more effective authority verification.

[0134] It should be noted that the above description is only provided as an example and should not be construed as limiting the present invention.

[0135] Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments are implemented as a program (computer program) executed by a computer data processing device. When the computer program is executed, the above method provided by the present invention can be implemented. Moreover, the computer program can be stored in a computer-readable storage medium, which can be a readable storage medium such as a disk, an optical disk, a ROM, a RAM, or a storage array composed of multiple storage media, such as a disk or tape storage array. The storage medium is not limited to centralized storage, and can also be distributed storage, such as cloud storage based on cloud computing.

[0136] Compared with the existing technology, the present invention can provide more accurate data basis for account authentication service by storing authorized account association information and periodically updating the authorized account association information; establish an effective account authentication service, and call the stored authorized account association information to perform account authentication service and authenticate access requests, which can reliably ensure the security of data in the Hive data warehouse, realize more effective identity authentication and permission verification process, and effectively prevent user misoperation and malicious operation.

[0137] Furthermore, by taking the approach of periodic data updates, the problem of online account authentication failure caused by inconsistency between the authorized account association information in the account authentication module and the database information in the metadata database can be effectively avoided; by adding a local database (or local MAP) to the metadata database, not only can it be ensured that when there is a problem with accessing the metadata database, the relevant data in the local database is used for authentication, but the access pressure on the metadata database can also be greatly reduced; by adding an access transition period and an account whitelist corresponding to the access transition period, and increasing the whitelist judgment step, it can be ensured that all accounts are authenticated and permission management is achieved during the access transition period; by calling the account authentication service, identity authentication is performed on access requests from different clients to the database server and the metadata server, which can limit malicious access by illegal users using non-upgraded clients and achieve a more effective identity authentication process; it can realize user group information for multiple data requests in the same connection, and can distinguish user group information of clients with different connections, thereby enabling more effective permission verification.

[0138] Example 2

[0139] The following describes a system embodiment of the present invention, which can be used to implement the method embodiment of the present invention. Details described in the system embodiment of the present invention should be considered supplementary to the above-described method embodiment; details not disclosed in the system embodiment of the present invention can be implemented with reference to the above-described method embodiment.

[0140] Reference Figure 2 、 Figures 4 to 7The present invention also provides an access control system 500 for a data warehouse, wherein the data warehouse is used to provide database access services and metadata access services, and the metadata is used to describe the database in the data warehouse.

[0141] like Figure 6 As shown, the access control system 600 includes: a storage module 601, which is used to store authorized account association information and update the authorized account association information by periodically obtaining metadata, wherein the authorized account association information includes authorized account information and its corresponding database information; an establishment module 602, which is used to establish an account authentication service independent of the database access service and the metadata access service in the data warehouse to call the stored authorized account association information; a receiving module 603, which is used to receive an access request from a client; a calling module 604, which is used to call the account authentication service to identify and process the account information (in this example, the name and password) in the access request based on the stored authorized account association information to perform identity authentication; a verification module 605, which is used to return the user request that has passed the identity authentication to the database access service and the metadata access service for permission verification.

[0142] Specifically, the Hive data warehouse includes a database access service and a metadata access service, and includes a first access entry and a second access entry corresponding to the database access service and the metadata access service. Figure 4 .

[0143] More specifically, because different data has different confidentiality levels, the data in the Hive data warehouse is assigned different permissions (including the actions users can take on the data) based on the confidentiality level and permitted actions. For example, some data is accessible to all users, while other data is accessible to only a limited number of users. Still other data is only viewable by a select few, with some users allowed to download and save, others allowed to delete or add data, and so on. In other words, different accounts (i.e., different users) are granted different permissions, thus creating authorized accounts.

[0144] The Hive data warehouse of the present invention is equipped with an authentication module, which is used to provide account authentication services. By calling the account authentication service to perform identity authentication, the security of data can be more reliably guaranteed, and the user's misoperation and malicious operation can be effectively prevented.

[0145] Specifically, an authentication module (such as a local MAP) is used to store the authorized account association information.

[0146] More specifically, the authorized account information includes the account number, account name, password, and database information corresponding to the account. The database information includes the data table where the data is located, the database to which the data table belongs, the database name, the data table name, the data table properties, the location of the data table where the data is located, etc.

[0147] Therefore, by storing the authorized account association information and periodically updating the authorized account association information, a more accurate data basis can be provided for the account authentication service.

[0148] For example, when the access request is a metadata access request, the account information (account name and password) in the access request is passed to the metadata access service through the user group information, and the account authentication service is called to perform identity authentication.

[0149] Optionally, the transmitting the account information (eg, account name and password) in the access request to the metadata access service via the user group information includes: adding a specific identifier to a prefix of the user group information.

[0150] It should be noted that, since the user group information and specific identification in this embodiment are the same as those in Embodiment 1, their description is omitted.

[0151] For example, when the access request is a metadata access request, the execution event of the metadata access request is monitored through a hook function, and permission verification is performed when the execution event is performed.

[0152] Specifically, the permission verification for database tables and databases is added to the permission verification of the original Metastore.

[0153] like Figure 7 As shown, in another example, the access control system 600 also includes a setting module 701, that is, splitting the calling module 604 into a calling module 604 and a setting module 701, and the setting module 701 is used to set the access transition period of the data warehouse and the account whitelist corresponding to the access transition period.

[0154] Specifically, during the access transition period, when the access request does not include an account name and password, it is determined whether the account information in the access request is in the whitelist. If so, the access request is treated as an access request for a public account, and the database access service and metadata access service directly perform identity authentication and permission management.

[0155] Optionally, source information corresponding to a public account is retrieved from the audit log of the data warehouse on a regular basis, and a prompt message is sent to the public account based on the source information.

[0156] Specifically, at the end of the transition period, the whitelist is deleted, and identity authentication and rights management are performed on all access requests to the database access service and the metadata access service. In other words, at the end of the transition period, identity authentication and rights management are enabled for all access requests.

[0157] It should be noted that the access transition period, the account whitelist corresponding to the access transition period, and the steps of whitelist judgment in this embodiment are the same as the access transition period, the account whitelist corresponding to the access transition period, and the steps of whitelist judgment in Example 1. Therefore, their description is omitted.

[0158] Therefore, by adding an access transition period and an account whitelist corresponding to the access transition period, and adding a whitelist judgment step, it is possible to ensure normal access to the old access method during the access transition period, that is, the transition period is compatible with the old method, so as to effectively avoid affecting the online production environment.

[0159] It should be noted that, in Example 2, descriptions of the same parts as in Example 1 are omitted.

[0160] Those skilled in the art will appreciate that the modules in the above system embodiments may be distributed in the system as described, or may be modified accordingly and distributed in one or more systems different from the above embodiments. The modules in the above embodiments may be combined into one module or further divided into multiple submodules.

[0161] Compared with the existing technology, the present invention can provide more accurate data basis for account authentication service by storing authorized account association information and periodically updating the authorized account association information; establish an effective account authentication service, and call the stored authorized account association information to perform account authentication service and authenticate access requests, which can reliably ensure the security of data in the Hive data warehouse, realize more effective identity authentication and permission verification process, and effectively prevent user misoperation and malicious operation.

[0162] Furthermore, by taking the approach of periodic data updates, the problem of online account authentication failure caused by inconsistency between the authorized account association information in the account authentication module and the database information in the metadata database can be effectively avoided; by adding a local database (or local MAP) to the metadata database, not only can it be ensured that when there is a problem with accessing the metadata database, the relevant data in the local database is used for authentication, but the access pressure on the metadata database can also be greatly reduced; by adding an access transition period and an account whitelist corresponding to the access transition period, and increasing the whitelist judgment step, it is possible to ensure normal access to the old access method during the access transition period, that is, the transition period is compatible with the old method, so as to effectively avoid affecting the online production environment; by calling the account authentication service, identity authentication is performed on access requests from different clients to the database server and the metadata server, which can limit malicious access by illegal users using non-upgraded clients and achieve a more effective identity authentication process; it can realize user group information for multiple data requests in the same connection, and can distinguish user group information of clients with different connections, thereby enabling more effective permission verification.

[0163] Example 3

[0164] The following describes a computer device embodiment of the present invention. This computer device can be considered a specific physical implementation of the method and apparatus embodiments of the present invention described above. Details described in the computer device embodiment of the present invention should be considered supplementary to the above-described method or apparatus embodiments; details not disclosed in the computer device embodiment of the present invention can be implemented with reference to the above-described method or apparatus embodiments.

[0165] Figure 8 is a structural diagram of a computer device according to an embodiment of the present invention, the computer device includes a processor and a memory, the memory is used to store a computer executable program, when the computer program is executed by the processor, the processor executes Figure 1 method.

[0166] like Figure 8 As shown, the computer device is implemented as a general-purpose computing device. The processor may be one or multiple processors working in concert. The present invention also does not exclude distributed processing, meaning that the processors may be dispersed across different physical devices. The computer device of the present invention is not limited to a single entity but may also be the sum of multiple physical devices.

[0167] The memory stores a computer executable program, generally a machine-readable code, which can be executed by the processor to enable the computer device to perform the method of the present invention, or at least some steps in the method.

[0168] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0169] Optionally, in this embodiment, the computer device further includes an I / O interface for exchanging data with external devices. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus structures.

[0170] It should be understood that Figure 8 The computer device shown is merely an example of the present invention. The computer device of the present invention may also include elements or components not shown in the above examples. For example, some computer devices also include display units such as screens, while others include human-computer interaction elements such as buttons and keyboards. As long as the computer device is capable of executing a computer-readable program stored in its memory to implement the method of the present invention or at least some of the steps of the method, it is considered a computer device within the scope of the present invention.

[0171] Figure 9 Schematic diagram of a computer program product according to an embodiment of the present invention. Figure 9 As shown, a computer program product stores a computer executable program, and when the computer executable program is executed, the above-mentioned method of the present invention is implemented. The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0172] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0173] Through the above description of the implementation mode, it is easy for those skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), or it can be distributed and stored on a network, as long as it enables a computer device to execute the method according to the present invention.

[0174] The specific embodiments described above further illustrate the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or computer equipment, and various general-purpose devices can also implement the present invention. The above description is merely a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A data warehouse access control method, characterized in that: include: Storing authorized account association information and updating the authorized account association information by periodically acquiring metadata, wherein the authorized account association information includes authorized account number information and its corresponding database information; Establish an account authentication service independent of the database access service and metadata access service to call the stored authorization account association information; Responding to the client's access request; Calling the account authentication service to identify and process the account information in the access request according to the stored authorized account association information to perform identity authentication; The request that passes the identity authentication is returned to the database access service and the metadata access service for authority verification.

2. The data warehouse access control method according to claim 1, characterized in that: When the access request is a metadata access request, the account information in the access request is passed to the metadata access service through the user group information, and the account authentication service is called to perform identity authentication; Optionally, a specific identifier is added to the prefix of the user group information.

3. The data warehouse access control method according to claim 1, characterized in that: The authority verification includes: When the access request is a metadata access request, the execution event of the metadata access request is monitored through a hook function, and permission verification is performed when the execution event is performed.

4. The data warehouse access control method according to claim 3, characterized in that: The access request is an access request to the Metastore of the Hive data warehouse, and the permission verification includes: adding permission verification for the database table and the database to the permission verification of the original Metastore.

5. The data warehouse access control method according to claim 1, characterized in that: Also includes: Setting an access transition period for the data warehouse and an account whitelist corresponding to the access transition period; During the access transition period, when the access request does not include an account name and password, determine whether the account information in the access request is in the whitelist. If so, treat the access request as an access request for a public account, and do not perform identity authentication and permission management on the access request for the public account.

6. The data warehouse access control method according to claim 5, characterized in that: Also includes: The source information corresponding to the public account is regularly retrieved from the audit log of the data warehouse, and a prompt message is sent to the public account based on the source information.

7. The data warehouse access control method according to claim 5, characterized in that: At the end of the transition period, the whitelist is deleted, and identity authentication and authority management are performed on all access requests to the database access service and the metadata access service.

8. An access control system for a data warehouse, characterized in that: include: A storage module is used to store the authorized account association information and update the authorized account association information by periodically acquiring metadata, wherein the authorized account association information includes the authorized account number information and its corresponding database information; Establishing a module for establishing an account authentication service in the data warehouse that is independent of the database access service and the metadata access service to call the stored authorized account association information; A receiving module, used for responding to an access request from a client; A calling module, used to call the account authentication service to identify and process the account information in the access request according to the stored authorized account association information to perform identity authentication; The verification module is used to return the user request that has passed the identity authentication to the database access service and the metadata access service for authority verification.

9. A computer device, wherein: The computer equipment includes: processor; and, A memory storing computer executable instructions, wherein when the executable instructions are executed, the processor executes the data warehouse access control method according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the access control method for a data warehouse according to any one of claims 1 to 7 is implemented.

Images