Security design and architecture for multi-tenant HADOOP clusters

Abstract

This application discloses a security design and architecture for multi-tenant Hadoop clusters. In one embodiment, in a multi-tenant Hadoop cluster comprising multiple tenants and multiple applications, a method for identifying, naming, and creating a multi-tenant directory structure in the multi-tenant Hadoop cluster may include: (1) identifying multiple groups of a directory structure selected from groups consisting of a superuser group, multiple tenant groups, and at least one application group; (2) creating a valid directory for each of the groups; (3) adding each of the multiple users to one of the multiple tenant groups and the application group; (4) creating a tenant directory and a home directory for the users; and (5) assigning an owner, group owner, default permissions, and extended access control lists to the tenant directory and the home directory.

Description

[0001] This application is a divisional application of Chinese patent application 201780044056.9, filed on May 23, 2017, entitled "Security Design and Architecture for Multi-tenant Hadoop Clusters".

[0002] Related applications

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 340,284, filed May 23, 2016, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0004] This invention relates in general to the security design and architecture for multi-tenant Hadoop clusters. Background Technology

[0005] Multitenancy is an architecture in which a single instance of a software application can serve multiple clients or tenants. Multitenancy can be economical because software development, update costs, and maintenance costs are shared. An example of multitenancy is disclosed in U.S. Patent Application Publication No. 2010 / 0005055, the disclosure of which is incorporated herein by reference in its entirety. Summary of the Invention

[0006] This invention discloses a security design and architecture for multi-tenant Hadoop clusters. In one embodiment, in a multi-tenant Hadoop cluster comprising multiple tenants and multiple applications, a method for identifying, naming, and creating a multi-tenant directory structure in the multi-tenant Hadoop cluster may include: (1) one of the tenants or applications identifying multiple groups of a directory structure comprising a superuser group, multiple tenant groups, and at least one application group; (2) one of the tenants or applications creating a valid directory for each of the groups; (3) one of the tenants or applications adding each of the multiple users to one of the multiple tenant groups and application groups; (4) one of the tenants or applications creating a tenant directory and a home directory for the users; and (5) one of the tenants or applications assigning an owner, group owner, default permissions, and extended access control lists to the tenant directory and the home directory.

[0007] In one embodiment, the directory structure can be an HDFS directory structure.

[0008] In one embodiment, the application group may include an application function group and an application human user group.

[0009] In one embodiment, users added to a tenant group can access the tenant's shared resources.

[0010] In one embodiment, users added to an application group can access the tenant's application resources and shared resources.

[0011] In one embodiment, the shared resources of the first tenant are isolated from the shared resources of the second tenant.

[0012] In one embodiment, the first tenant cannot access the resources of the second tenant.

[0013] In one embodiment, the shared resources of the first application are isolated from the shared resources of the second application.

[0014] According to another embodiment, in a multi-tenant Hadoop cluster including multiple tenants and multiple applications, a method for providing security to HDFS applications in the multi-tenant Hadoop cluster may include: (1) one of the tenants or applications authenticating a client process against the client; (2) one of the tenants or applications receiving a request from the client that includes at least one of a session ticket and a temporary session key from a key distribution center; and (3) one of the tenants or applications authenticating the client based on at least one client authorization and at least one of a session ticket and a temporary session key.

[0015] In one embodiment, the key distribution center may be a Kerberos key distribution center.

[0016] In one embodiment, the client can authenticate with the key distribution center by providing a username and password.

[0017] In one embodiment, the method may further include identifying at least one group of clients selected from a group that includes a superuser group, multiple tenant groups, and at least one application group.

[0018] According to another embodiment, in a multi-tenant Hadoop cluster including multiple tenants and multiple applications, a method for providing security for applications in the multi-tenant Hadoop cluster may include: (1) one of the tenants or applications identifies multiple groups and roles for multiple tenants and identifies cross-references between said roles and at least one valid directory group; (2) one of the tenants or applications creates a valid directory for each of the groups; (3) one of the tenants or applications adds each of the multiple users to the tenant group and the application group; (4) one of the tenants or applications creates a base directory for the application mode; and (5) one of the tenants or applications assigns roles and privileges to the tenants.

[0019] In one embodiment, the application could be Apache HIVE or Cloudera Impala.

[0020] In one embodiment, roles and privileges can be based on at least one tenant application requirement.

[0021] In one embodiment, the method may further include a tenant or an application authenticating a user with a username and password; and a tenant or an application authorizing the user to use role-based fine-grained authorization.

[0022] According to another embodiment, in a multi-tenant Hadoop cluster including multiple tenants and multiple applications, a method for providing security for applications in the multi-tenant Hadoop cluster may include: (1) one of the tenants or applications identifies multiple HBase namespaces and groups that have permissions to HBase namespaces; (2) one of the tenants or applications creates a valid directory for each of the groups; (3) one of the tenants or applications creates an HBase namespace to satisfy at least one tenant requirement; and (4) one of the tenants or applications creates a default role for each application that has at least one privilege.

[0023] In one embodiment, at least one privilege can control access to the application.

[0024] In one embodiment, the method may further include one of the tenants or applications using an access control list to authorize client actions.

[0025] In one embodiment, the method may also include granting client permissions using role-based access control within a tenant or application. Attached Figure Description

[0026] To more fully understand the invention, its objects and advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which:

[0027] Figure 1 A multi-tenant cluster according to one embodiment is described;

[0028] Figure 2 A hierarchical structure of multitenant objects is described according to one embodiment;

[0029] Figure 3 A logical group structure in an effective catalog according to one embodiment is described;

[0030] Figure 4 A user-to-group mapping is described according to one embodiment;

[0031] Figure 5 A method for creating a directory structure for HDFS according to one embodiment is described;

[0032] Figure 6 An HDFS directory structure according to one embodiment is described;

[0033] Figure 7 An ownership and access control list of directories in an HDFS directory structure according to one embodiment is shown;

[0034] Figure 8 A method for providing security for HDFS in a multi-tenant cluster, according to one embodiment, is described;

[0035] Figure 9 A method for providing security for HIVE or Impala in a multi-tenant cluster, according to one embodiment, is described;

[0036] Figure 10 The structure of the HIVE database and parent directory according to one embodiment is described;

[0037] Figure 11 A method for providing security for HBase in a multi-tenant cluster, according to one embodiment, is described;

[0038] Figure 12 The responsibilities of the Sentry DBA and tenant DBA according to one embodiment are described, as well as how to grant the necessary privileges to the appropriate groups; and

[0039] Figure 13 This illustrates how a tenant DBA, according to one embodiment, can grant the necessary privileges to the appropriate group. Detailed Implementation

[0040] According to embodiments, multi-tenant clusters enable multiple tenants to securely share a common set of cluster resources using, for example, strong authentication and authorization policies instead of physical separation. In one embodiment, the system and method can achieve some or all of the following: (1) minimal or no negative impact on existing service level agreements (SLAs); (2) no violation of existing security requirements and security policies; (3) no need to disclose the presence of each tenant residing in the multi-tenant environment; (4) the ability to audit users' actual data access (human and functional); (5) the ability to report current permissions on datasets; and (6) the ability to run multiple concurrent applications with guaranteed resources.

[0041] refer to Figure 1 A multi-tenant cluster is disclosed according to one embodiment. In one embodiment, the multi-tenant computing environment 100 may include a multi-tenant Hadoop cluster 110. In one embodiment, the Hadoop cluster may be provided for development, production, quality assurance, etc.

[0042] In one embodiment, the multi-tenant Hadoop cluster 110 can store, process, and analyze large amounts of data. The multi-tenant Hadoop cluster 110 can support various services including Hive, Yarn, Impala, HBase, and HDFS.

[0043] In one embodiment, multiple utility nodes 1201, 1202, etc., can be provided, and one or more tenants 1301, 1302, ... 130 can be supported. n For example, utility node 1201 can support tenant 1301, while utility node 1202 can support tenants 1302 and 1303. Any suitable support configuration can be used as needed and / or desired.

[0044] In one embodiment, utility nodes 1201, 1202, etc., can serve as interfaces between the multi-tenant Hadoop cluster 110 and external networks. In another embodiment, utility nodes 1201, 1202, etc., can be used to run client applications and cluster management tools. Utility nodes 1201, 1202, etc., can also be used as temporary storage areas for data transferred to the multi-tenant Hadoop cluster 110.

[0045] In one embodiment, a “tenant” can be any business entity within an organization, vertical, or application that can pay for, be responsible for, etc., resources in the multi-tenant computing environment 100. Exemplary characteristics of a tenant may include: (1) multiple tenants reside in a single large cluster (e.g., a Hadoop cluster); (2) each tenant may have one or more applications; (3) each application may have specific requirements for the cluster to meet its needs; and (4) different types of user accounts may exist in the cluster. For example, there may be service accounts (e.g., accounts that can be used to run services (e.g., Hadoop services), personal accounts (e.g., accounts that can be used by people who access the cluster), and functional accounts (e.g., accounts that can be used to run applications). A personal account may have access to multiple or a single tenant / multiple or a single application.

[0046] refer to Figure 2 According to one embodiment, a multi-tenant hierarchical structure is disclosed. As shown in the figure, each tenant 1301, 1302 may have one or more applications (e.g., application 1 (App 1), application 2 (App 2), ... application n (App n)) that can be accessed by one or more users (e.g., user 1-user 13).

[0047] refer to Figure 3 and Figure 4The document discloses a method for group mapping for tenant groups according to one embodiment. In one embodiment, the group mapping policy can be a combination of group mapping based on a Lightweight Directory Access Protocol (or LDAP) and static bindings. LDAP groups can be used for both individual and functional accounts, as users can manage them within an active directory (“AD”). Static bindings can be used for service accounts (e.g., HDFS, Hive, Impala, etc.), as these are limited and can be managed within a Hadoop configuration file.

[0048] Table I below shows the different groups for multi-tenant security requirements according to one embodiment:

[0049]

[0050] Table I

[0051] In one embodiment, adding a user to a tenant group will allow the user to access only the tenant's shared resources, but not application resources. Adding a user to an application group will allow the user to access only application resources, but not the tenant's shared resources.

[0052] refer to Figure 3 It shows the logical group structure in the valid directory, and in Figure 4 The image shows the mapping from users to groups. Figure 4 In the diagram, solid lines indicate that a user belongs to the group, and dashed lines indicate that a user belongs to the management group. As shown in the diagram, user 1 is a member of the tenant group, application group 1, and management group; user 2 is a member of the tenant group and application group N; and user 3 is a member of the tenant group, application group 2, and application group N.

[0053] refer to Figure 5 It provides a method for creating a directory structure for HDFS according to one embodiment.

[0054] In step 510, groups can be identified according to a naming convention. In one embodiment, the exemplary naming conventions in Table II can be used:

[0055]

[0056] Table II

[0057] It should be noted that this naming convention is merely exemplary and any suitable naming convention may be used as needed and / or desired.

[0058] In step 520, a valid directory group can be created, and users can be added to the group. Any user loaded representing the application can be part of at least two groups—the tenant group and the application group.

[0059] In step 530, a tenant directory and a home directory can be created for the user. In one embodiment, a specific HDFS directory structure can be used. Figure 6 and Figure 7 An exemplary directory structure is shown below, and details are presented in Table III:

[0060]

[0061] Table III

[0062] like Figure 6 and Figure 7 As shown, in one embodiment, a tenant can support multiple applications that may require complete isolation. Furthermore, data can be shared between multiple applications under the same tenant.

[0063] In step 540, owner, group owner, default permissions, and extended access control lists (ACLs) can be assigned to the directory. As indicated in Table III, in one embodiment, ownership of the tenant, application, and shared directory, along with the ACLs, is modeled so that the tenant cannot delete its own base directory but can still access the data to meet its application requirements.

[0064] Figure 7 This shows the ownership and ACL settings of the user directories under / user. In one embodiment, a user will not have the ability to delete their own home directory, but will be able to access all data within their home directory.

[0065] In one embodiment, security can be provided. In one embodiment, fine-grained control over file permissions can be implemented using, for example, ACLs and Sentry. In one embodiment, ACLs can be applied depending on (multiple) security requirements. Sentry can be used for structured data managed using, for example, Hive, Impala, etc.

[0066] Furthermore, HDFS can be configured to allow communication from users belonging to a tenant's valid directory group. In one embodiment, by default, HDFS in a multi-tenant environment can reject all communication except from users belonging to the allowed valid directory groups.

[0067] refer to Figure 8This document discloses a method for providing security to applications in a Hadoop multi-tenant cluster according to one embodiment. In one embodiment, the application may be a Hadoop Distributed File System or HDFS. HDFS is designed to store very large files with streaming data access patterns and is intended to run on commodity hardware clusters. HDFS is a logical collection of files, partitioned and distributed across several blocks, with its metadata stored in the "Namenode".

[0068] Hadoop supports two operating modes for user authentication, which can be specified by the attribute `hadoop.security.authentication`. First, simple authentication can be used, where the identity of the client process can be determined by the host operating system. Second, Kerberos authentication can be used to authenticate users.

[0069] In one embodiment, once a user's username is determined, a list of groups can be determined using a group mapping service, which can be configured via the hadoop security.group.mapping property. Exemplary group matching options may include: static binding (i.e., qualifying user-to-group mappings in a Hadoop configuration file); shell-based group mapping (i.e., resolving groups on the master node (Namenode / Resource Manager) using the command "bash -c groups" or "net group"); and LDAP group mapping (i.e., directly connecting to an LDAP server to resolve the group list).

[0070] In one embodiment, multi-tenant cluster 810 can process requests from client 830, which has authenticated with a Key Distribution Center (KDC) 840 (such as a Kerberos KDC) using a username and password from client 830. KDC 840 can verify the username / password via user directory 820. If authentication is successful, KDC can provide client 830 with one or more session tickets and / or temporary session keys. Client 830 can then provide the session ticket / temporary session key to multi-tenant cluster 810. In one embodiment, multi-tenant cluster 810 can use the temporary session key to authenticate client 830 and fulfill requests from the client based on client 830's authorization.

[0071] refer to Figure 9The document discloses a method for providing security to an application in a Hadoop multi-tenant cluster, according to another embodiment. In one embodiment, the application may be an Apache Hive data warehouse infrastructure for providing data summaries, queries, and analysis. The Hive Query Language (HiveQL) includes a subset of SQL and some extensions that can be used to run analytical queries on large datasets stored in HDFS. Hive can structure data into easily understood database concepts such as tables, columns, rows, and partitions.

[0072] In another embodiment, the application could be Cloudera Impala. Impala is an open-source massively parallel processing (MPP) SQL query engine for storing data in computer clusters running Apache Hadoop. Impala integrates with the Apache Hive metastore database to share databases and tables between the two components. This strong integration with Hive and compatibility with HiveQL syntax allows users to create tables, issue queries, load data, and more using either Impala or Hive.

[0073] In one embodiment, security for Hive and Impala can be implemented using authentication (e.g., using Kerberos supported by LDAP or user / password authentication) and authorization (e.g., using Sentry for role-based fine-grained authorization). In one embodiment, a predefined directory structure can isolate the structured data stored by each tenant. A base directory can be created for each tenant, where the tenant can store multiple structured data schemas in HDFS.

[0074] In step 910, groups and roles can be identified according to naming conventions. In one embodiment, cross-references between roles and valid directory groups can be identified. In one embodiment, the exemplary naming conventions of Table IV can be used:

[0075]

[0076] Table IV

[0077] It should be noted that this naming convention is merely exemplary and any suitable naming convention may be used as needed and / or desired.

[0078] In step 920, a valid directory group can be created, and users can be added to the group. Any user representing the application can be part of at least two groups—a tenant group and an application group.

[0079] In step 930, a base directory for structured data can be created for the schema. In one embodiment, a specific directory structure can be used. Figure 10 An exemplary directory structure is shown below, and details are displayed in Table V:

[0080]

[0081] Table V

[0082] Figure 10 This shows the structure of the HIVE database along with its parent directory. It shows that each tenant's HIVE base directory is located in its own tenant directory. Table V shows the owners, groups, and permissions on the hive schema directory.

[0083] Base directory ownership can be granted to users and groups within the HIVE schema. Fine-grained permissions for tenant users on these HIVE schemas can be controlled using settings such as those configured in Sentry. Sentry permissions can be automatically translated into extended ACLs on HDFS files and directories.

[0084] In one embodiment, the attributes of a subdirectory may include permissions inherited from its parent, ACL, and Sentry policies.

[0085] In step 940, privileged roles can be set. In one embodiment, the roles and the privileges associated with the roles can be driven by requirements from one or more of the tenant's applications.

[0086] In one embodiment, each tenant can be granted a DBA role with all privileges over its own schema. This allows tenants to manage the authorization of their schema themselves. DBA roles can be assigned to specific valid directory groups, where only service accounts with additional privileges are onboarded as members.

[0087] In one embodiment, after roles are created, these roles can be assigned to valid directory groups provided in previous steps 910 and 920.

[0088] In one embodiment, the assignment can result in privileges being automatically assigned to files and directories as HDFS ACLs. This provides consistent authorization for data, whether accessed from Hive, Impala, or directly through the HDFS interface.

[0089] refer to Figure 11 According to another embodiment, a method for providing security to applications in a Hadoop multi-tenant cluster is disclosed. In one embodiment, the application may be HBase, an open-source NoSQL database that provides real-time read / write access to those large datasets.

[0090] In one embodiment, Kerberos can be used for authentication, allowing HBase servers and clients to securely identify themselves using HDFS, ZooKeeper, and each other. Access control lists, or ACLs, can be used to authorize various operations (READ, WRITE, CREATE, ADMIN) by column, column family, and column family qualifier. HBase ACLs can be granted and revoked for both users and groups.

[0091] In one embodiment, the HBase security model can use RBAC (role-based access control), whereby access permissions can be stored at the metadata layer and applied when a user attempts to access a table or column.

[0092] In one embodiment, HBase can use the same set of permissions for ACLs at the system, namespace, table, and column family levels. Permissions granted at higher levels can be inherited by objects at lower levels. For example, if a group is granted namespace-level read permissions, members of that group can read all tables in that namespace.

[0093] Groups can be assigned privileges that effectively grant tenant-level ownership of namespaces. Tenant administrators / database administrators can control who can access which tables within their namespace. In one embodiment, different groups at each level of the environment might be needed to manage HBase structured data.

[0094] Table VI below illustrates the different group types and responsibilities according to one embodiment.

[0095]

[0096] Table VI

[0097] Table VII below describes the goals and responsibilities of different types of users or groups according to one embodiment. Table VII is merely exemplary; access models can be created based on the tenant's application requirements.

[0098]

[0099]

[0100] Table VII

[0101] In step 1110, the names of HBase namespaces and groups, along with their permissions, can be identified according to a naming convention. In one embodiment, the exemplary naming convention in Table VII can be used:

[0102] Table VIII

[0103] It should be noted that this naming convention is merely exemplary and any suitable naming convention may be used as needed and / or desired.

[0104] In step 1120, a valid directory group can be created, and users can be added to the group. In one embodiment, users who own and manage namespaces can be identified, and Hadoop groups can be created in the valid directory as needed.

[0105] In step 1130, a namespace that meets the tenant's requirements can be created.

[0106] In step 1140, a default role can be created for each application, and the default role can have one or more read and / or write privileges.

[0107] The following non-restrictive examples are provided.

[0108] Example 1: HDFS Scenario—Onboarding two tenant machines to a multi-tenant Hadoop cluster—Retail Banking and Enterprise Marketing.

[0109] Each of these tenants has multiple applications that they want to run on a multi-tenant cluster and that have specific security requirements:

[0110]

[0111] Table IX

[0112] The requirements are as follows: Add two tenants along with the applications listed in Table IX, and provide the credit card application with read-only access to the data in the personal banking application.

[0113] In one embodiment, a Hadoop supergroup can be a prerequisite for the solution described below. The process is outlined as follows: (1) identifying valid directory groups for tenants and applications; (2) creating valid directory groups for tenants; (3) creating necessary directories and home directories for users; (4) assigning owners, groups, and permissions to directories; and (5) ACLs.

[0114] First, groups in table X can be identified according to the naming conventions discussed above.

[0115] type name Hadoop Group tenant Retail banking business ND-MT-RTLBNK tenant Corporate Marketing ND-MT-CORPMKT app credit card ND-MT-RTLBNK-CC / ND-MT-RTLBNK-CC-F app Personal banking services ND-MT-RTLBNK-PBANK / ND-MT-RTLBNK-PBANK-F app mortgage ND-MT-RTLBNK-MTG / ND-MT-RTLBNK-MTG-F app sports ND-MT-CORPMKT-CAMP / ND-MT-CORPMKT-CAMP-F app Overall Marketing ND-MT-CORPMKT-GMKT / ND-MT-CORPMKT-GMKT-F

[0116] Table X

[0117] Next, you can create a valid directory group.

[0118] Next, you can add the user to the group.

[0119] Next, the HDFS directory in Table XI can be created using superuser privileges with appropriate permissions.

[0120]

[0121]

[0122] Table XI

[0123] Example 2: Adding two tenants (retail banking and enterprise marketing) to a multi-tenant Hadoop cluster. Furthermore, as shown in Table XII, each of these tenants may have multiple applications with specific security requirements that they wish to run on the multi-tenant cluster.

[0124]

[0125] Table XII

[0126] Requirements: (1) Identify the user who owns and manages the database and create Hadoop groups in the valid directory if necessary; (2) Create a DBA role for each tenant and attach it to the specific database; and (3) Create a default role for each application with read privileges, write privileges, or both.

[0127] In one embodiment, a Sentry management group can be set up, which can be named "ND-MT-ADMIN".

[0128] When using Sentry to manage users, you can set up DBA roles for each tenant and assign roles to Hadoop groups. For example, Table XIII below shows tenant DBAs and their associated roles.

[0129]

[0130]

[0131] Table XIII

[0132] The following table XIV shows an exemplary Hadoop group and the name of the database for each tenant application in this solution.

[0133]

[0134] Table XIV

[0135] In one embodiment, to establish a "Retail Banking" tenant account and database, the following steps can be used: (1) create a role named "mt_rtlbnk_dba" using authorization options; (2) grant the role "mt_rtlbnk_dba" to the group "ND-MT-DIG-DBA"; and (3) grant full permissions to the role "mt_rtlbnk_dba" to the URI (the location in HDFS used to store the tenant's structured data). This allows one or more tenant-level DBAs to access data in the specified HDFS location, and any user in the group ND-MT-RTLBNK-DBA can act as a tenant DBA for retail banking.

[0136] Next, (4) create the database “db_rtlbnk_cc”; (5) grant all privileges to the tenant DBA role “mt_rtlbnk_dba” for the database “db_rtlbnk_cc”; (6) create the database “db_rtlbnk_pbank”; and (7) grant all privileges to the tenant DBA role “mt_rtlbnk_dba” for the database “db_rtlbnk_pbank”.

[0137] In one embodiment, to set up a "Corporate Marketing" tenant DBA role and database, the following steps can be used: (1) create a role named "mt_corpmkt_dba" with granted privileges; (2) grant the role "mt_rsk_dba" to the group "ND-MT-CORPMKT-DBA"; and (3) grant full permissions to the role "mt_corpmkt_dba" to the URI (the location in HDFS used to store the tenant's structured data). This allows one or more tenant-level DBAs to access data in the specified HDFS location, and any user in the group ND-MT-CORPMKT-DBA can work as a tenant DBA for risk purposes.

[0138] Next, (4) create the database “db_corpmkt_camp”; (5) grant all privileges of the database “db_corpmkt_camp” to the tenant dba role “mt_corpmkt_dba”; (6) create the database “db_corpmkt_gmkt”; and (7) grant all privileges of the database “db_corpmkt_gmkt” to the tenant dba role “mt_corpmkt_dba”.

[0139] In one embodiment, each tenant may have application-specific security requirements, and they require three roles in each application—read-only, write, and all. These roles can be the tenant's application-level sense roles and privileges.

[0140] In one embodiment, the following roles can be created, for example, through Sentry Management: roles for the "Credit Card" application ("mt_rtlbnk_cc_insert"; "mt_rtlbnk_cc_read"; and "mt_rtlbnk_cc_all"); and roles for the "Personal Banking" application ("mt_rtlbnk_pbank_insert"; "mt_rtlbnk_pbank_read"; and "mt_rtlbnk_pbank_all"). In one embodiment, the Sentry DBA can create any additional roles required by the application's security requirements.

[0141] Figure 12 The responsibilities of the Sentry DBA and tenant DBA according to one embodiment are described, as well as how to grant the necessary privileges to the appropriate groups.

[0142] Figure 13 This illustrates how a tenant DBA, according to one embodiment, can grant the necessary privileges to the appropriate group.

[0143] It should be noted that although several embodiments have been disclosed, the embodiments disclosed herein are not mutually exclusive.

[0144] In the following text, the general aspects of implementing the systems and methods of the present invention will be described.

[0145] For example, the system of the present invention, or a portion thereof, may be in the form of a “processor” (such as a general-purpose computer). As used herein, the term “processor” should be understood to include at least one processor using at least one memory. At least one memory stores an instruction set. Instructions may be stored permanently or temporarily in one or more memories of the processor. The processor executes the instructions stored in the one or more memories for processing data. The instruction set may include various instructions for performing one or more specific tasks (such as those described above). Such an instruction set for performing a specific task may be characterized as a program, a software program, or simply as software.

[0146] In one embodiment, the processor may be a dedicated processor.

[0147] As noted above, the processor executes instructions stored in one or more memories to process data. For example, such data processing may be in response to commands from one or more users of the processor, in response to previous processing, in response to a request from another processor, and / or any other input.

[0148] As noted above, the processor used to implement the present invention can be a general-purpose computer. However, the processor described above can also utilize any of a variety of other technologies, including dedicated computers, computer systems (including, for example, microcomputers, minicomputers or mainframes, programmable microprocessors, microcontrollers), peripheral integrated circuit elements, CSIC (customer application-specific integrated circuits) or ASIC (application-specific integrated circuits) or other integrated circuits, logic circuits, digital signal processors, programmable logic devices (such as FPGAs, PLDs, PLAs or PALs), or any other means or arrangement of means capable of implementing the steps of the process of the present invention.

[0149] The processor used to implement this invention can utilize a suitable operating system. Therefore, embodiments of this invention may include processors running the following operating systems: iOS, OS X, Android, and Microsoft Windows. TM Operating systems, Unix operating systems, Linux operating systems, Xenix operating systems, IBM AIX TM Operating system, Hewlett-Packard UX TM Operating system, Novell Netware TM Operating system, Sun Microsystems Solaris TM Operating system, OS / 2 TM Operating system, BeOS TM Operating systems, Macintosh operating system, Apache operating system, OpenStep TM Operating system or other operating systems or platforms.

[0150] It should be understood that, in order to practice the method of the present invention as described above, the processor and / or memory of the processor do not necessarily need to be physically located in the same geographical location. That is, each of the processor and memory used by the processor can be located in geographically different locations and connected to communicate in any suitable manner. Additionally, it should be understood that each of the processor and / or memory can consist of different physical devices. Therefore, the processor does not necessarily have to be a single device in one location, and the memory does not necessarily have to be another single device in another location. That is, it is contemplated that the processor can be two devices in two different physical locations. The two different devices can be connected in any suitable manner. Additionally, the memory can include two or more memory portions in two or more physical locations.

[0151] To further illustrate, as described above, the processing is performed by various components and various memories. However, it should be understood that, according to another embodiment of the invention, a process performed by two different components as described above can be performed by a single component. Furthermore, a process performed by one different component as described above can be performed by two different components. Similarly, according to another embodiment of the invention, memory storage performed by two different memory portions as described above can be performed by a single memory portion. Furthermore, memory storage performed by one different memory portion as described above can be performed by two memory portions.

[0152] Furthermore, various technologies can be used to provide communication between various processors and / or memories, and to allow the processors and / or memories of this invention to communicate with any other entity; i.e., to obtain further instructions or to access and use remote memory storage. Such technologies for providing this communication may include networks, the Internet, intranets, extranets, LANs, Ethernet, wireless communications via cell towers or satellites, or any client-server system providing communication. Such communication technologies can use any suitable protocol, such as TCP / IP, UDP, or OSI.

[0153] As described above, instruction sets can be used in the processing of this invention. Instruction sets can take the form of programs or software. For example, software can take the form of system software or application software. For example, software can also take the form of a collection of individual programs, program modules within a larger program, or a portion of a program module. The software used can also include modular programming in the form of object-oriented programming. The software tells the processor how to respond to the data being processed.

[0154] Furthermore, it should be understood that the instructions or instruction sets used in the implementation and operation of this invention can be in a suitable form so that a processor can read the instructions. For example, the instructions forming a program can be in the form of a suitable programming language, which is converted into machine language or object code to allow one or more processors to read the instructions. That is, a compiler, assembler, or interpreter is used to convert lines of programming code or source code written in a particular programming language into machine language. Machine language is binary-encoded machine instructions specific to a particular type of processor (i.e., a particular type of computer). Computers understand machine language.

[0155] Any suitable programming language can be used according to various embodiments of the invention. Illustratively, the programming languages ​​used may include, for example, assembly language, Ada, APL, Basic, C, C++, COBOL, dBase, Forth, Fortran, Java, Modula-2, Pascal, Prolog, REXX, Visual Basic, and / or JavaScript. Furthermore, it is not necessary to combine a single type of instruction or a single programming language with the operation of the system and method of the invention. Rather, any number of different programming languages ​​can be used as needed and / or desired.

[0156] Furthermore, the instructions and / or data used in the practice of this invention can utilize any compression or encryption technology or algorithm as needed. An encryption module can be used to encrypt data. Additionally, for example, a suitable decryption module can be used to decrypt files or other data.

[0157] As described above, the present invention can be illustratively implemented in the form of a processor, including, for example, a computer or computer system including at least one memory. It should be understood that, as needed, the instruction set (i.e., software) that enables the computer operating system to perform the above operations can be contained on any of one or more various media. Additionally, the data processed by the instruction set can also be contained on any of one or more various media. That is, the specific medium (i.e., the memory in the processor) used to store the instruction set and / or data used in the present invention can, for example, take any of various physical forms or transmission methods. Illustratively, the medium can be in the form of: paper, paper-transparent film, optical disc, DVD, integrated circuit, hard disk, floppy disk, optical disc, magnetic tape, RAM, ROM, PROM, EPROM, wire, cable, optical fiber, communication channel, satellite transmission, memory card, SIM card or other telemetry, and any other data medium or data source readable by the processor of the present invention.

[0158] Furthermore, one or more memories used in the processor implementing this invention can be in any of various forms to allow the memory to store instructions, data, or other information as needed. Therefore, the memory can be in the form of a database to store data. For example, the database can use any desired file arrangement, such as a flat file arrangement or a relational database arrangement.

[0159] In the systems and methods of this invention, various "user interfaces" can be utilized to allow a user to connect to one or more processor interfaces used to implement the invention. As used herein, a user interface includes any hardware, software, or a combination of hardware and software used by the processor that allows the user to interact with the processor. For example, a user interface may take the form of a dialog box. A user interface may also include any of the following: a mouse, a touchscreen, a keyboard, a keypad, a voice reader, a voice recognizer, a dialog box, a menu box, a list, a checkbox, a toggle switch, a button, or any other means that allows the user to receive information about the operation of the processor while processing a set of instructions and / or providing information to the processor. Therefore, a user interface is any means that provides communication between a user and a processor. For example, information provided by the user to the processor through the user interface may be in the form of commands, data selections, or some other input.

[0160] As described above, the user interface is utilized by a processor that executes a set of instructions to process data for the user. The user interface is typically used by the processor to interact with the user in order to transmit or receive information from the user. However, it should be understood that, in some embodiments of the systems and methods according to the invention, the human user does not actually need to interact with the user interface used by the processor of the invention. Instead, it is also contemplated that the user interface of the invention can interact with another processor (rather than a human user), i.e., to transmit and receive information. Therefore, other processors can be characterized as users. Furthermore, it is contemplated that the user interface utilized in the systems and methods of the invention can interact partially with one or more other processors while also interacting partially with a human user.

[0161] It will be readily understood by those skilled in the art that the present invention is readily available and applicable. Many embodiments and adaptations of the invention, as well as many variations, modifications and equivalent arrangements, will be apparent or reasonably suggested in the invention and in its foregoing description, without departing from the spirit and scope of the invention.

[0162] Therefore, although the invention has been described in detail herein with respect to exemplary embodiments thereof, it should be understood that this disclosure is merely illustrative and exemplary and is intended to provide a disclosure of how the invention can be implemented. Consequently, the foregoing disclosure is not intended to be construed as limiting the invention or to otherwise exclude any other such embodiments, adaptations, variations, modifications, or equivalent arrangements.

Claims

1. A method for providing security to HDFS applications in a multi-tenant Hadoop cluster, comprising: In a single multi-tenant, multi-node Hadoop cluster that includes multiple nodes, multiple tenants, and multiple applications: One of the tenants or applications is the client authentication client process; One of the tenants or applications receives a request from the client, the request including at least one of a session ticket and a temporary session key from the key distribution center; One of the tenants or applications authenticates the client based on at least one client authorization and at least one of the session ticket and the temporary session key; as well as One of the tenants or applications assigns an owner, group owner, default permissions, and extended access control lists (EACLs) to multiple tenant directories and multiple home directories, wherein the EACLs prevent unauthorized tenants from accessing resources in the tenant directories and the home directories.

2. The method according to claim 1, wherein the key distribution center is a Kerberos key distribution center.

3. The method of claim 1, wherein the client authenticates with the key distribution center by providing a username and password before the tenant or application receives a request from the client.

4. The method according to claim 3, further comprising: One of the tenants or applications is determined to be selected from at least one group of the clients, which includes a superuser group, multiple tenant groups, and at least one application group.

5. A method for providing security for applications in a multi-tenant Hadoop cluster, comprising: In a single multi-tenant, multi-node Hadoop cluster that includes multiple nodes, multiple tenants, and multiple applications: One of the tenants or applications identifies multiple groups and roles for multiple tenants and identifies cross-references between the roles and at least one valid directory group; One of the tenants or applications creates a valid directory for each of the groups; One of the tenants or applications adds each of the multiple users to the tenant group and the application group; One of the tenants or applications creates a base directory for the application mode; One of the tenants or applications uses an Extended Access Control List (EACL) to assign roles and privileges to the tenant; and One of the tenants or applications creates a tenant shared space for sharing data between applications within the tenant, wherein the EACL prevents unauthorized tenants from accessing the tenant shared space.

6. The method of claim 5, wherein the application is Apache HIVE.

7. The method of claim 5, wherein the application is Cloudera Impala.

8. The method of claim 5, wherein the roles and privileges are based on at least one tenant application requirement.

9. A method for providing security for HBase in a multi-tenant Hadoop cluster, comprising: In a single multi-tenant, multi-node Hadoop cluster that includes multiple nodes, multiple tenants, and multiple applications: One of the tenants or applications has multiple HBase namespaces and groups with permissions to HBase namespaces; One of the tenants or applications creates a valid directory for each of the groups; One of the tenants or applications creates the HBase namespace to satisfy at least one tenant's requirements; One of the tenants or applications creates a default role for each application with at least one privilege; One of the tenants or applications creates a tenant shared space for sharing data between applications within the tenant, wherein an extended access control list prevents unauthorized tenants from accessing the tenant shared space.

10. The method of claim 9, wherein the at least one privileged control controls access to the application.

11. The method of claim 9, further comprising: One of the tenants or applications uses an access control list to authorize client actions.

12. The method of claim 9, further comprising: One of the tenants or applications uses role-based access control to grant permissions to the client.

Images