Secure data access in virtual data processing

Abstract

This describes secure data access in virtual data processing. An example includes instructions for: receiving a request from an application on a compute node of a compute cluster within a virtual data processing environment for a user to access a secure data source, the virtual data processing environment including multiple secure data sources accessible to the compute nodes of the virtual compute cluster; retrieving credentials in the current application context and forwarding the credentials for authentication; verifying the credentials through a credential authority; and, upon successful credential verification, authenticating the user at the secure data source and establishing a connection to the secure data source.

Description

Technical Field

[0001] This application relates to secure data access in virtual data processing. Background Technology

[0002] In large-scale virtual data processing environments, challenges arise regarding compute and storage resources. For this reason, compute and storage separation has become increasingly common, as it helps decouple compute and storage resources from each other.

[0003] In a virtual storage system, there may be multiple different computing and storage resources to support a variety of different applications. For example, computing resources may include the operation of a virtual computing cluster to support different applications, and storage resources (which may also be referred to as data sources here) may include multiple different storage resources that can be accessed by certain applications.

[0004] However, when multiple storage resources are available, separating compute and storage can lead to complications. In such an architecture, storage sources can be protected using a variety of different security authentication solutions, depending on the nature of the data source. Because different types of data sources may have different security authentication mechanisms, applications running in the compute cluster are typically required to support the specific authentication mechanism of each data source before accessing data in the data source. Attached Figure Description

[0005] The examples described herein are shown in the accompanying drawings by way of example rather than by way of limitation, and similar reference numerals in the accompanying drawings refer to similar elements.

[0006] Figure 1 This is an illustration of a virtual data processing environment that provides secure access to multiple data sources, based on some examples;

[0007] Figure 2 These are illustrations of the deployment of elements that provide secure data access in a large virtual data processing environment, based on some examples.

[0008] Figure 3A These are illustrations of systems that provide secure data access, based on some examples;

[0009] Figure 3B These are illustrations of systems that provide secure data access, based on some examples;

[0010] Figure 4 This is a flowchart illustrating a process for providing secure data access in a virtual data processing environment, based on some examples;

[0011] Figure 5 This is a flowchart illustrating a process for providing secure data access in a virtual data processing environment, based on some examples;

[0012] Figure 6 An example system for providing secure data access in a virtual data processing environment is described, based on several examples; and

[0013] Figure 7 An example system for providing secure data access in a virtual data processing environment is described, based on several examples. Detailed Implementation

[0014] In virtual data processing, a system may include a computing architecture and a separate storage architecture to provide separation and decoupling between these elements. In large systems, there may be many different secure data sources, each protected by a specific authentication technology associated with it. In operation, applications typically access multiple different data sources. In the implementation of multiple data sources with different authentication standards, this usually requires successfully establishing authentication with each relevant data source.

[0015] However, in conventional technologies, the authentication requirement for multiple different data sources necessitates compatibility with each relevant authentication technology. This requirement can significantly increase the complexity of application development and operation, as each application can include implementations of multiple security technologies to enable successful access to the data source. Further operational complexity may arise because in large systems, applications can be distributed across multiple nodes, and therefore multiple instances can simultaneously authenticate multiple data sources.

[0016] In some examples, to support applications running on virtual computing clusters to access multiple different secure data sources simultaneously, while also reducing the complexity and cost of application development and processing, the apparatus or system includes an authentication support system. The authentication support system provides information exchange to perform authentication for multiple data sources and enables applications to access these data sources without requiring the applications themselves to support each authentication standard. In this way, standard applications can operate on systems with independent compute and storage architectures without needing to modify the applications to be compatible with each applicable authentication standard. For example, changes can be made to a data lake implementation without rewriting one or more applications, because such applications can access multiple secure data sources without needing to conform to the authentication mechanism of each such data source.

[0017] In some examples, the authentication support system includes a data broker deployed to each node of the compute cluster to handle data source access requests. Additionally, the authentication support system includes a data daemon deployed independently of the data broker (such as deployed outside the container or virtual machine that includes the data broker), which authenticates users and authenticates the data source through the data source's authentication mechanism. As used herein, "data broker" refers to a computer program, such as a computer program installed on a node of the compute cluster, and "data daemon" or "daemon" refers to a computer program that runs as a background process in a multitasking operating system.

[0018] In some examples, data brokers and data daemons can work together to authenticate users, then use the data source's authentication mechanism for further authentication, thereby reducing the authentication requirements and overhead for applications accessing multiple data sources in operation. For instance, a data broker can implement an API (Application Programming Interface) for data source access. When a user or service of an application running on a node in a compute cluster attempts to access a remote data source via the API, the node's data broker intercepts the request and forwards it to the associated data daemon.

[0019] The data daemon then combines the access request with the user's authentication. Following successful user authentication, the data daemon acts on behalf of the user to authenticate the data source through its authentication mechanism. After successful authentication, the data daemon serves the original request by accessing the API provided by the appropriate data source, and then responds to the user's original request based on the data source's response.

[0020] In some examples, the system only requires authentication credentials (such as Kerberos credentials) to be accepted, and the application itself can be unkerberized under the authentication protocol, allowing applications running on the computing cluster to access secure data sources through user authentication. Furthermore, the HDFS (Hadoop Distributed File System) data source concept has been extended to a general secure data source, allowing access to any of multiple different data sources. HDFS is a distributed file system for processing large datasets running on commodity hardware, while Hadoop is a distributed computing framework developed by the Apache Software Foundation.

[0021] Figure 1This is an illustration of a virtual data processing environment that provides secure access to multiple data sources, based on several examples. As shown, the virtual data processing environment 100 includes a virtual computing cluster 120, which includes one or more host systems. In the example shown, the virtual computing cluster 120 includes multiple compute nodes (shown as compute node 1 to compute node N) to support applications within the virtual data processing environment 100. Each compute node can be a container or a virtual machine (VM). As used herein, a “virtual machine” refers to the virtualization or emulation of a computer system, and a “container” refers to an executable unit of software in which application code is packaged or bundled with associated configuration files, libraries, and dependencies.

[0022] Environment 100 may include decoupled compute and storage resources. In this architecture, environment 100 may include multiple secure data sources 140, shown as data source 1 to data source N, where each secure data source may include multiple data sources of different types, each including applicable security authentication technologies. Authentication technologies for secure data sources may include Kerberos, MapR tickets, OAuth for Google Cloud Storage, access keys for AWS S3, etc. Therefore, in normal operation, applications running on compute nodes in the virtual compute cluster 120 need to support specific authentication mechanisms for each of the multiple secure data sources 140 accessed by the application in operation. In a specific example of Kerberos authentication technology, if an application running in the virtual compute cluster 120 cannot be kerberized, the unkerberized application will not be able to access kerberized data sources using normal technologies.

[0023] In some examples, the process or system supports applications running in the virtual computing cluster 120 to access multiple different secure data sources 140 in a separated compute and storage architecture. In some examples, access includes multiple accesses to data sources within a single operation, which can include accesses at the same time or similar times, thereby supporting jobs distributed across multiple nodes in the virtual computing cluster. Figure 1 As shown, one or more hosts in the computing cluster include security support 160 for multiple different security data sources 140, thereby allowing applications to access multiple data sources without requiring the application to be compatible with each authentication mechanism associated with a data source. In a specific example, security support 160 can allow an application that has not been kerberized to access a kerberized data source. Furthermore, security support 160 also allows applications to access unkerberized data sources, further simplifying operations.

[0024] In some examples, security support 160 provides one or more data agents and data daemons in the compute and storage architecture, where security support includes the operation of the application's data agent and the operation of the data daemon associated with the relevant host on the compute node where the application is stored. For example, the data agent may be deployed to each node of a virtual compute cluster 120, where each node corresponds to a virtual machine or container. Furthermore, the data daemon is deployed separately from the data agent and independent of the compute cluster. For example, the compute cluster may include containers or virtual machines with data agents, and the data daemon is deployed outside the containers or virtual machines. The data daemon may be deployed in a separate compute node on a physical host or otherwise separately from the data agent. In some examples, the data agent deployed to each node of the virtual compute cluster handles data source access requests, and the data daemon is deployed separately to authenticate users and authenticate the data source through the data source's authentication mechanism.

[0025] Figure 2 This is an illustration of the deployment of elements providing secure data access in a large virtual data processing environment, based on some examples. In some examples, the virtual data processing system 200 includes one or more host computing systems 230. Figure 2 The diagram shows n hosts, i.e., host 1 to host n. System 200 also includes a set of separate secure data sources 240, wherein the secure data sources may include different types of storage technologies, and each data source includes applicable security authentication technologies to protect access to the data source. In this diagram, the data sources may include, but are not limited to, Hadoop Distributed File System (HDFS) 242 with a Kerberos Key Distribution Center (KDC); MapR storage cluster 244; Google Cloud Storage cluster 246; and AWS (Amazon Web Services) S3 (Simple Storage System) cloud storage. The security authentication technologies of the secure data sources 240 may include Kerberos for HDFS system 242, MapR tickets for MapR cluster 244, OAuth for Google Cloud cluster 246, access keys for AWS S3 248, etc. In addition, secure data sources 240 may include applicable APIs, such as Hadoop File System API, MapR File System API, Google Cloud Storage API, AWS S3 API, or other data source APIs.

[0026] Each host computing system 230 may include one or more compute nodes, which may include one or more containers, virtual machines, or both, wherein the compute nodes on the host system form a virtual computing cluster 220. In this particular example, host 1 is shown as including j containers, indicated as containers 1 to J, and host N is shown as including k containers, indicated as containers 1 to K (in... Figure 2In this context, n, j, and k can be represented as any positive integer greater than 1. As further explained, system 200 includes an authentication center (…). Figure 2 The instruction (KDC 250) is used to provide user authentication, such as regarding Figure 3A and Figure 3B Further described. Additionally, system 200 may include management (Mgt 255) for managing configuration information for virtual computing cluster 220 and secure data source 240.

[0027] In some examples, users or services in virtual computing cluster 220 can access multiple secure data sources 240 through the operation of data brokers and data daemons (for the purpose of providing authentication services). For example... Figure 2 As shown, each compute node (such as containers 1 to J on host 1 and containers 1 to K on host N) may include an associated data agent 266, which is deployed to handle data source access requests. Additionally, data daemons (such as data daemons 263 or 264) are deployed separately to authenticate users and secure data sources 240 through the data source's authentication mechanism.

[0028] In this way, the solution decouples user authentication and data source authentication operations. For example, in Hadoop Distributed File System (HDFS) pass-through to DataTap security, both the application in the compute cluster and the HDFS data source are kerberized before accessing DataTap. Conversely, in this solution, only Kerberos credentials are required, and the application can be unkerberized, allowing unkerberized compute clusters to access the secure data source through user authentication. Furthermore, the HDFS data source is extended to general-purpose secure data sources, such as MapR clusters or cloud storage, and the data source can use any other security mechanism without requiring kerberization.

[0029] In one example, the data broker and data daemon can work together to enable authentication for different data sources simultaneously. Internally, the data daemon maintains connection and security authentication information for each data resource and creates a separate connection context for each user in the virtual computing cluster to access any data resource.

[0030] Figure 3AThis is an illustration of a system providing secure data access, based on several examples. In a particular example, system 300, which may include multiple host nodes, includes a virtual computing cluster 320 with multiple compute nodes (such as containers or virtual machines). In this illustration, the virtual computing cluster 320 includes at least a first compute node (container A) and a second compute node (container B) to support applications to run on system 300. System 300 also includes at least one authentication center (including a Kerberos Key Distribution Center (KDC) (shown as KDC 350) or other centers) and one or more data sources (including data source 340 shown) for access by supported applications. Figure 3A In this system, each compute node includes a data agent, such as data agent A associated with container A and data agent B associated with container B, to handle data source access requests. Additionally, system 300 includes one or more data daemons, such as data daemon A, which are deployed individually to authenticate users and authenticate data sources through the authentication mechanism of each data source.

[0031] In some examples, a data broker can implement an API (Application Programming Interface) for data source access. This API can include, but is not limited to, the Hadoop Distributed File System (HDFS) API, which models the contents of the file system as a set of paths, which are either directories, symbolic links, or files. The S3 Object API is another example of a data source access API. When a user or service of an application running on a node in a compute cluster attempts to access a remote data source via the Hadoop File System API, the request is intercepted, such as by the node's data broker, which forwards the request to the associated data daemon. In addition to the request, the data broker also forwards the current user context information associated with the request.

[0032] In some examples, the data daemon verifies user identity based at least in part on received user context information. User information such as forwarded POSIX (Portable Operating System Interface developed by the IEEE Computer Society) information is insufficient for user authentication because the forwarded POSIX user might be switched to a POSIX user instead of the actual POSIX user running the application on the compute node. In some examples, user context information includes Kerberos credentials or delegation tokens to allow the data daemon to successfully authenticate the user based on such user context information. As used herein, a delegation token is a security token issued to a delegate to act as the user.

[0033] Following successful user authentication, the data daemon acts on behalf of the user to authenticate the data source through its authentication mechanism. After successful user authentication, the data daemon serves the original request by accessing the API provided by the appropriate data source, and then responds to the user's original request based on the data source's response.

[0034] In some examples, a request for authentication credentials (such as Kerberos credentials) from a credential authority (e.g., KDC 350) is made, where the credentials are received at a compute node (such as container A) in a virtual compute cluster. The container supports the application, and a data broker is deployed within the container. When an application attempts to access a remote data source, the associated data broker deployed in the container is invoked. The associated data broker retrieves the user credentials within the current application context, and the user credentials are forwarded to the data daemon for authentication. When a user or service issues a command to retrieve user credentials, authentication credentials (such as Kerberos credentials from KDC 350) can be received in the relevant compute node (such as container A). For example, the "kinit" command can be issued to retrieve Kerberos credentials for authentication. The kinit command creates a ticket cache for storing Kerberos credentials. When the first application in container A invokes data broker A, data broker A can retrieve the credentials and forward them to data daemon A to establish a connection with the corresponding data source (data source 340). For example, in Kerberos operations:

[0035] When computing cluster 320 has already undergone Kerberization, data agent A is invoked in the Kerberos security environment, enabling data agent A to directly retrieve Kerberos credentials based on the current Kerberos context.

[0036] When compute cluster 320 has not undergone Kerberization, data broker A detects that the existing Kerberos context is unavailable. Data broker A can proactively search for Kerberos credentials in the following ways:

[0037] — Add a custom configuration property to indicate the location of Kerberos credentials. For example, add the property "bdfs.kerberos.${USERNAME}.ticket.cache" to the core-site.xml configuration file.

[0038] —If the credentials are not found in the custom configuration, you can further search for them in the default ticket cache directory (such as / tmp / krb5cc_${UID}) or in environment variables (such as KRB5CCNAME). (KRB5CCNAME refers to the default Kerberos credential cache name.)

[0039] In large-scale virtual data processing deployments, jobs can be distributed across multiple compute cluster nodes for parallel processing, and therefore each such node may need to access the same data source at the same time. To avoid providing authentication credentials (Kerberos credentials in this example) on each node, the Kerberos credentials 365 can be placed on shared storage 360, and the attribute ("bdfs.kerberos.${USERNAME}.ticket.cache") can be used to indicate the shared location. The shared Kerberos credentials can then be accessed by the data broker on each node.

[0040] In some examples, data daemon A can directly verify forwarded Kerberos credentials via KDC350 in the following ways:

[0041] If a dedicated service principal is configured in the KDC 350, the data daemon A can use forwarded Kerberos credentials to obtain service tickets.

[0042] If no dedicated service principal is configured in KDC 350, the data daemon updates existing Kerberos credentials through KDC 350.

[0043] After successfully verifying the Kerberos credentials, daemon A abstracts the Kerberos principal from the Kerberos credentials and retrieves the username from the Kerberos principal.

[0044] After successfully verifying the identity of the current user, data daemon A can act as the verified user to connect to data source 340 for authentication, where the authentication mechanism depends on the data source.

[0045] When configuring a data source for application access, in addition to the necessary data source type and the hostname / IP location of the data source itself, an authentication mechanism and associated authentication secret are also provided for the data source. In certain specific examples:

[0046] If the data source is HDFS storage, the authentication secret is the associated keytab file.

[0047] If the data source is MapR storage, the authentication secret is the associated MapR ticket file.

[0048] If the data source is GCS (Google Cloud Storage) storage, the authentication secret is the associated JSON (JavaScript Object Notation) key file.

[0049] If the data source is AWS S3, the authentication secret includes the associated access key (access key ID and secret access key) stored.

[0050] In this operation, data daemon A can further forward the current user information to data source 340 during the authentication process, which can be used to combat possible user impersonation.

[0051] After successful authentication and the establishment of a connection between the data daemon and the data source, confirmation that the connection has been successfully established is provided. In some examples, data access can proceed on behalf of the user.

[0052] Figure 3B This is an illustration of a system providing delegated data access based on some examples. In a particular example, system 300 again includes a virtual computing cluster 320 having multiple compute nodes (containers or virtual machines), such as a first compute node (container A) and a second compute node (container B), to support applications to run on system 300. System 300 also includes at least one KDC (shown as KDC 350) or other credential element, and one or more data sources for access by supported applications, including data source 340 shown. Each compute node includes a data broker, such as data broker A associated with container A and data broker B associated with container B, to handle data source access requests. Furthermore, system 300 includes one or more data daemons, including data daemon A and data daemon B, which are deployed individually to authenticate users and authenticate data sources via a per-data-source-specific authentication mechanism. System 300 may also include, for example, shared storage 360 ​​to store credentials (such as Kerberos credentials 365) to support jobs distributed across multiple compute cluster nodes for parallel processing and to avoid the need to provide Kerberos credentials on each node.

[0053] In some examples, the system can provide extended delegation tokens to reduce interaction with KDC 350 when virtual computing cluster 320 has already undergone Kerberization. This is after successful authentication of data source 340, as described above. Figure 3B The application can retrieve the delegation token via the file system API (Hadoop Distributed File System (getDelegationToken() API)) based on the connection established with the data source 340.

[0054] Data daemon A can generate and return a dedicated delegation token based on the currently verified user, timestamp, and authentication secret of the data source. The delegation token includes at least two elements: an identifier and a password, which can be generated as follows:

[0055] "Identifier: The identifier includes current data source service information (such as hostname or IP address), user information, timestamp (current and expired), and sequence number.

[0056] "Password: The password may include an HMAC (Hash-based Message Authentication Code) signature based on an identifier and an authentication secret from a data source."

[0057] In one example, a delegation token can be forwarded to container B, such as through the Hadoop delegation token framework. When data agent B is invoked by an application, data agent B obtains a delegation token from its user context and forwards the delegation token to data agent B to establish a connection with the corresponding data source.

[0058] In this example, after data daemon B receives the delegation token, it verifies the token. Daemon B calculates the HMAC (using the identifier and the authentication secret of the data source) and compares the result with the password. If these values ​​do not match, this indicates that the token has been tampered with, and data daemon B will reject the request. If the calculated HMAC value matches the password, data daemon B can continue to verify whether the service matches the data source and check whether the timestamp has expired.

[0059] After successful verification of the delegation token, data daemon B can obtain the username from the verification token and, on behalf of the verified user, authenticate the data source based on the data source's authentication information. After successful authentication and the establishment of the connection between data daemon B and the data source, data daemon B confirms that the connection has been successfully established and can continue data access.

[0060] In another example, although the Hadoop Distributed File System API does not conform to the POSIX API, the permission models for files and directories used in the Hadoop File System share elements of the POSIX model. Cloud storage such as GCS and AWS S3 typically do not provide POSIX-style access control and therefore do not offer user / group-based ACLs.

[0061] For cloud storage, the data daemon can utilize the metadata of data objects provided by the cloud storage to store user / group information. When accessing cloud storage through a data broker and the data daemon, the data daemon can also verify the current user, such as information about... Figure 3A and Figure 3BAs described above, when creating a directory or file, the data daemon can add verified user and associated group information to the metadata of the corresponding object representing the directory or file in cloud storage.

[0062] In this example, when an authenticated user accesses directories and files, the data daemon can apply the POSIX permission model and ACLs (Access Control Lists). When listing directories or reading files, the data daemon can retrieve metadata representing the corresponding object in cloud storage; and then authorize the request based on the authenticated user and associated group information, as well as the file owner / group information stored in the metadata.

[0063] In another alternative example, if data source 340 also supports Kerberos authentication, such as a Kerberized HDFS, user authentication can be delegated to the data source. In this way, the data daemon does not need to authenticate users directly through the KDC, but can instead authenticate the data source using forwarded Kerberos credentials. If the virtual compute cluster 320 is already Kerberized, and data source 340 supports generating and validating delegation tokens, delegation token generation / validation can also be delegated to the data source.

[0064] In this scenario, if the KDC used to kerberize the computing cluster and the KDC used to kerberize the data source are different, then a cross-realm Kerberos trust can be configured between these two KDCs.

[0065] Figure 4 This is a flowchart illustrating a process for providing secure data access in a virtual data processing environment, based on some examples. In process 400, a request for a security credential (such as a Kerberos credential) from a credential authority (e.g., a KDC) is made, and the credential is received 410 at a first compute node in a virtual computing cluster, where the compute node supports the application, and where a first data agent is deployed in the compute node.

[0066] When an application requests access to a remote data source, the data agent deployed on the first compute node is invoked (412), and the data agent intercepts the request. The first data agent retrieves credentials in the current application context and forwards them to the first data daemon (deployed separately from the first data agent) to establish a connection with the corresponding data source (416). If the compute cluster has been Kerberized, data agent A will be invoked in the Kerberos security environment, and the data agent can directly retrieve Kerberos credentials based on the current Kerberos context. If the compute cluster has not been Kerberized, the data agent actively searches for Kerberos credentials by adding custom configuration properties to indicate the location of the Kerberos credentials. If the credentials are not found in the custom configuration, the data agent performs a search in the default directory. In some examples, to avoid providing credentials on every node of the compute cluster, the credentials can be placed on shared storage so that the data agent on all nodes of the compute cluster can access them.

[0067] In some examples, the data daemon directly verifies the forwarded user credentials 418 through the credential authority. If a dedicated service principal is configured in the KDC, the data daemon uses the forwarded credentials (such as Kerberos credentials) to obtain a service ticket. If no dedicated service principal is configured in the KDC, the data daemon uses the KDC to update the Kerberos credentials.

[0068] After successfully verifying the Kerberos credentials, the data daemon abstracts from the credentials the identity to which the credential authority can assign tickets (the Kerberos principal is the identity to which Kerberos can assign tickets), and retrieves the username 420 from the identity (Kerberos principal).

[0069] After successfully verifying the current user's identity, the first data daemon can act as the verified user to establish a connection with the data source, allowing the application to access data source 422. The nature of the authentication mechanism depends on the data source. When configuring access to each data source, in addition to the necessary data source type and hostname / IP to connect to, the data source's authentication mechanism and associated authentication secret are provided for authentication. For example, if the data source is HDFS, the authentication secret is the associated keytab file; if the data source is MapR storage, the authentication secret is the associated MapR ticket file; if the data source is GCS (Google Cloud Storage), the authentication secret is the associated JSON key file; and if the data source is AWS S3, the authentication secret is the associated access key (access key ID and secret access key).

[0070] During authentication, the data daemon forwards the current user information to the data source to prevent potential impersonation 424. After successful authentication and the establishment of a connection between the data daemon and the data source, confirmation that the connection has been successfully established is provided 426. The data daemon can then proceed with the requested data access 428.

[0071] Figure 5 This is a flowchart illustrating a process for providing delegated data access in a virtual data processing environment, based on some examples. In some examples, the application may be distributed across multiple nodes in a virtual computing cluster, which may include a first computing node with a first data proxy and a second computing node with a second data proxy. In process 500, after successful authentication of the data source, as... Figure 4 As shown, the application can retrieve the delegation token 510 via the file system API based on the connection established with the data source.

[0072] The first data daemon generates and returns a dedicated delegation token based on the current verified user, timestamp, and authentication secret 512 of the data source. The delegation token includes at least an identifier and a password, where the identifier may include current data source service information (such as hostname or IP address), user information, timestamp (current and expired), and sequence number, and the password may include an HMAC (Hash-Based Message Authentication Code) signature based on the identifier and the authentication secret of the data source. Process 500 may further include forwarding the delegation token to a second compute node 514, which includes a second data broker.

[0073] When the second data agent is invoked by the application, it obtains a delegation token from its user context and forwards it to the second data daemon to establish a connection with the corresponding data source 516. Upon receiving the delegation token, the second data daemon verifies it 518. During verification, the second data daemon calculates the HMAC using the identifier and the data source's authentication secret and compares the result with the password to determine if the values ​​match. If the values ​​do not match 520, this may indicate that the token has been tampered with, and the second data daemon rejects the access request 522. If the values ​​match, the second data daemon continues verification to determine if the requesting client or service matches the data source and checks if the timestamp has expired 524.

[0074] After successful verification of the delegation token, the second data daemon retrieves the username from the verified token and, on behalf of the verified user, authenticates the data source based on the data source's authentication information (526). Following successful authentication of the data source and the establishment of the connection between the second data daemon and the data source, the second data daemon confirms the connection has been successfully established (528) and continues with the requested data access (530).

[0075] Figure 6 Example systems for providing secure data access in a virtual data processing environment are depicted according to several examples. Example system 600 includes a non-transitory machine-readable medium 604 encoded with example instructions 610, 612, 616, 618, 620, 622, 624, 626, and 628 (collectively, instructions 610-628) executable by processing resource 602. In some implementations, system 600 can be used to execute... Figure 4 The process is 400.

[0076] Processing resource 602 may include a microcontroller, microprocessor, (multiple) central processing unit cores, ASIC (Application-Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and / or other hardware devices suitable for retrieving and / or executing instructions from machine-readable medium 604 to perform functions associated with various examples. Additionally or alternatively, processing resource 602 may include or be coupled to electronic circuitry or special-purpose logic for performing some or all of the functions of the instructions.

[0077] Machine-readable medium 604 can be any medium suitable for storing executable instructions, such as RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, hard disk drive, optical disk, etc. In some example implementations, machine-readable medium 604 can be a tangible, non-transitory medium. Machine-readable medium 604 can be located within system 600, in which case the executable instructions can be considered to be mounted or embedded on system 600. Alternatively, machine-readable medium 604 can be a portable (e.g., external) storage medium or part of an installation package.

[0078] As further described below, the machine-readable medium 604 can be encoded using a set of executable instructions 610-628. It should be understood that, in alternative implementations, some or all of the executable instructions and / or electronic circuitry included in one box may be included in different boxes shown in the figures, or in different boxes not shown in the figures. Some implementations of system 600 may include more than Figure 6 More or fewer instructions are shown.

[0079] When executed, instruction 610 causes processing resource 602 to request credentials (such as Kerberos credentials) from a credential authority (e.g., KDC) and receive the credentials at a first compute node in a virtual computing cluster, wherein the first compute node supports the application and wherein a first data broker is deployed in the compute node.

[0080] Instruction 612, when executed, causes processing resource 602 to invoke the first data broker when the application attempts to access a remote data source. Instruction 616, when executed, causes processing resource 602 to retrieve credentials in the current application context and forward them to the first data daemon to establish a connection with the corresponding data source. If the compute cluster has been Kerberized, the data broker will be invoked within the Kerberos security environment, and the data broker can directly retrieve Kerberos credentials based on the current Kerberos context. If the compute cluster has not been Kerberized, the data broker actively searches for Kerberos credentials by adding custom configuration properties to indicate the location of the Kerberos credentials. If the credentials are not found in the custom configuration, the data broker performs a search in the default directory. In some examples, to avoid providing credentials on every node of the compute cluster, the credentials can be placed on shared storage so that the data broker on all nodes of the compute cluster can access them.

[0081] When instruction 618 is executed, it causes processing resource 602 to directly verify the forwarded credentials for the user through the credential authority. If a dedicated service principal is configured in the KDC, the data daemon uses the forwarded credentials (such as Kerberos credentials) to obtain a service ticket. If no dedicated service principal is configured in the KDC, the data daemon uses the KDC to update the Kerberos credentials.

[0082] When instruction 620 is executed, it causes processing resource 602 to abstract the identity to which the credential issuing authority can assign tickets (the Kerberos principal is the identity to which Kerberos can assign tickets) from the credential after successful verification of the Kerberos credentials, and to obtain the username from the identity (Kerberos principal).

[0083] When instruction 622 is executed, it causes processing resource 602 to act as the authenticated user to connect to the data source for authentication after successfully verifying the identity of the current user. The authentication mechanism depends on the data source. When configuring each data source, in addition to the necessary data source type and hostname / IP required to connect to the data source, the data source's authentication mechanism and associated authentication secret are provided for authentication. For example, if the data source is HDFS, the authentication secret is the associated keytab file; if the data source is MapR storage, the authentication secret is the associated ticket file; if the data source is GCS (Google Cloud Storage), the authentication secret is the associated JSON key file; and if the data source is AWS S3, the authentication secret is the associated access key (access key ID and secret access key).

[0084] Instruction 624, when executed, causes processing resource 602 to forward the current user information to the data source during authentication to resolve potential impersonation. Instruction 626, when executed, causes processing resource 602 to provide the user or service with confirmation that the connection has been successfully established after successful authentication from the data source. Instruction 628, when executed, causes processing resource 602 to request the client or service to continue with the requested data access.

[0085] Figure 7 An example system for providing delegated data access in a virtual data processing environment is depicted, based on several examples. Example system 700 includes a non-transitory machine-readable medium 704 encoded with example instructions 710, 712, 714, 716, 718, 720, 724, 726, 728, and 730 (collectively referred to as instructions 710-730) executable by processing resource 702. In some implementations, system 700 can be used to execute... Figure 5 The process is 500.

[0086] The processing resources 702 and the machine-readable medium 704 can be as follows Figure 6 The description of processing resource 602 and machine-readable medium 604 is as described herein, and will not be repeated here. As further described below, machine-readable medium 704 may be encoded with a set of executable instructions 710-730. It should be understood that in alternative implementations, some or all of the executable instructions and / or electronic circuitry included in one box may be included in different boxes shown in the figures, or in different boxes not shown in the figures. Some implementations of system 700 may include more than Figure 7 More or fewer instructions are shown.

[0087] When instruction 710 is executed, it causes processing resource 702 to retrieve a delegation token for a virtual computing cluster via a file system API based on a connection to the data source established by the first data daemon. The virtual computing cluster includes nodes, such as a first computing node including a first data agent and a second computing node including a second data agent.

[0088] When instruction 712 is executed, it causes processing resource 702 to generate and return a dedicated delegation token based on the currently verified user, timestamp, and authentication secret of the data source. The delegation token includes at least an identifier and a password, where the identifier may include current data source service information (such as hostname or IP address), user information, timestamp (current and expired), and sequence number, and the password may include an HMAC (Hash-Based Message Authentication Code) signature based on the identifier and the authentication secret of the data source. When instruction 714 is executed, it causes processing resource 702 to forward the delegation token to a second compute node in the virtual computing cluster.

[0089] When instruction 716 is executed, it causes processing resource 702 to obtain a delegation token from the user context of the second data agent when the application calls the second data agent in the second compute node, and forward the delegation token to the second data daemon to establish a connection with the corresponding data source.

[0090] Instruction 718, when executed, causes processing resource 702 to verify the delegation token when the second data daemon receives it. Delegation token verification includes calculating the HMAC using the identifier and the authentication secret of the data source, and comparing the result with the password to determine if the values ​​match. Instruction 720, when executed, causes processing resource 702 to reject the access request if the comparison values ​​do not match. Instruction 724, when executed, causes processing resource 702 to continue verifying the delegation token after determining that the comparison values ​​match, to determine if the requesting client or service matches the data source, and to check if the timestamp has expired.

[0091] When executed, instruction 726 causes processing resource 702 to retrieve the username from the verified token after successful verification of the delegation token, and to authenticate the data source on behalf of the verified user based on the data source's authentication information. When executed, instruction 728 causes processing resource 702 to confirm that the connection has been successfully established after successful authentication and the establishment of the connection to the data source. When executed, instruction 730 causes processing resource 702 to request the client or service to continue with the requested data access.

[0092] The following clauses are further examples. Features can be applied anywhere in one or more examples. Various features of different examples can be combined differently, including some features and excluding others, to suit a variety of different applications. Examples can include subjects such as methods, means for performing actions of the methods, at least one machine-readable medium (such as a non-transitory machine-readable medium) including instructions that, when executed by a machine, cause the machine to perform the actions of the methods, or apparatus or systems for facilitating operation according to the examples described herein.

[0093] In the above description, numerous specific details have been listed for illustrative purposes to provide a comprehensive understanding of the described examples. However, it will be apparent to those skilled in the art that the examples can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form. Intermediate structures may exist between the illustrated components. Components described or shown herein may have additional inputs or outputs that are not stated or described.

[0094] Different examples can include different processes. These processes can be executed by hardware components or represented in computer programs or machine-executable instructions that can be used to cause general-purpose or special-purpose processors or logic circuits programmed with instructions to execute these processes. Alternatively, these processes can be executed by a combination of hardware and software.

[0095] Various examples may be provided as computer program products, which may include computer-readable media on which computer program instructions are stored, which can be used to program a computer (or other electronic device) to perform processing by one or more processors according to a particular example. Computer-readable media may include, but are not limited to: magnetic disks, optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic or optical cards, flash memory, or other types of computer-readable media suitable for storing electronic instructions. Furthermore, the examples may also be downloaded as computer program products, wherein the program can be transferred from a remote computer to the requesting computer. In some examples, a non-transitory computer-readable storage medium stores data thereon representing a sequence of instructions that, when executed by a processor, cause the processor to perform certain operations.

[0096] The process can be added to or removed from any of the methods described above, and information can be added to or subtracted from any of the described messages without departing from the basic scope of the present example. It will be apparent to those skilled in the art that many further modifications and adaptations can be made. The specific examples provided are not intended to limit the concept, but rather to illustrate it. The scope of these examples cannot be determined by the specific examples provided above, but only by the appended claims.

[0097] If element A is coupled to or is coupled with element B, then element A can be directly coupled to element B or indirectly coupled through, for example, element C. When the specification or claim states that component, feature, structure, process, or characteristic A causes component, feature, structure, process, or characteristic B, this means that A is at least partly responsible for B, but there may also be at least one other component, feature, structure, process, or characteristic that helps cause B. If the specification indicates that a component, feature, structure, process, or characteristic "may," "may," or "possibly" be included, then that particular component, feature, structure, process, or characteristic is not necessarily included. If the specification or claim refers to an element "a" or "an," this does not mean that only one element is described.

[0098] Examples are implementations. References to “example,” “an example,” “some examples,” or “other examples” in the specification indicate that a particular feature, structure, or characteristic described in conjunction with that example is included in at least some examples, but not necessarily in all examples. Different representations of “example,” “an example,” or “some examples” do not necessarily refer to the same example. It is noteworthy that in the foregoing description of examples, various features are sometimes combined in a single example, drawing, or description thereof, with the aim of simplifying the disclosure and aiding understanding of one or more of the various novel aspects. However, this method of disclosure should not be construed as reflecting an intention that the claimed example requires more features than expressly listed in each claim. Rather, as reflected in the appended claims, novel aspects are not present in all features of a single foregoingly disclosed example. Therefore, the claims are expressly incorporated herein, each claim being a standalone example.

Claims

1. A non-transitory computer-readable storage medium comprising instructions that, when executed, cause a system to: intercept, by a first data agent in a first virtual compute node, a first request from an application in the first virtual compute node, the first request for access to a secure data source for a requestor, the first virtual compute node being one of a plurality of virtual compute nodes of a compute cluster in a virtual data processing environment, the virtual data processing environment comprising a plurality of secure data sources accessible to the plurality of virtual compute nodes, the secure data sources being protected by a secure authentication technique; forward, by the first data agent, the first request to a first data daemon outside the first virtual compute node; verify, by the first data daemon, a credential from a current application context associated with the first request with a credential issuer; based on successful verification of the credential: authenticate, by the first data daemon, the requestor using the secure authentication technique for the secure data source, establish a connection with the secure data source for access to the secure data source by the application, and generate, by the first data daemon, a delegation token comprising a value based on an authentication secret of the secure data source; send the delegation token to a second virtual compute node of the plurality of virtual compute nodes; send, by a second data agent in the second virtual compute node, the delegation token to a second data daemon for a second request to access the secure data source intercepted by the second data agent; verify, by the second data daemon, the delegation token from the second data agent; and based on verification of the delegation token, authenticate, by the second data daemon, to the secure data source for the requestor that initiated the second request.

2. The non-transitory computer-readable storage medium of claim 1, wherein the plurality of secure data sources comprises a second secure data source, the second secure data source being protected by a second secure authentication technique, the second secure authentication technique being different from the secure authentication technique for the secure data source.

3. The non-transitory computer-readable storage medium of claim 2, wherein the secure authentication technique for the secure data source uses a first type of authentication secret, and the second secure authentication technique for the second secure data source uses a second type of authentication secret, the second type of authentication secret being different from the first type of authentication secret.

4. The non-transitory computer-readable storage medium of claim 1, wherein the first data agent implements an application programming interface (API), and the first request for access to the secure data source is made via the API.

5. The non-transitory computer-readable storage medium of claim 4, wherein the API comprises a Hadoop Distributed File System (HDFS) API. ​ 6. The non-transitory computer-readable storage medium of claim 1, wherein the credential is obtained from a shared storage that is accessible to each virtual compute node of the compute cluster.

7. The non-transitory computer-readable storage medium of claim 1, wherein the value in the delegation token comprises a signature, and wherein verifying the delegation token by the second data daemon comprises: computing, by the second data daemon, a signature based on the authentication secret of the secure data source, and comparing, by the second data daemon, the computed signature to the signature in the delegation token.

8. The non-transitory computer-readable storage medium of claim 1, further comprising instructions that, when executed, cause the system to: request the credential from the credential issuer, and receive the credential at the first virtual compute node.

9. The non-transitory computer-readable storage medium of claim 8, wherein the credential is a Kerberos credential, and the credential issuer is a Kerberos key distribution center (KDC).

10. The non-transitory computer-readable storage medium of claim 1, wherein each of the first virtual compute node and the second virtual compute node comprises a container or a virtual machine.

11. A method comprising: intercepting, by a first data agent in a first virtual compute node, a first request from an application in the first virtual compute node, the first request for access to a secure data source for a requestor, the first virtual compute node being one of a plurality of virtual compute nodes of a compute cluster in a virtual data processing environment, the virtual data processing environment comprising a plurality of secure data sources accessible to the plurality of virtual compute nodes, the secure data sources being protected by a secure authentication technique; forwarding, by the first data agent, the first request to a first data daemon outside the first virtual compute node; verifying, by the first data daemon, a credential from a current application context with a credential issuer, the current application context being associated with the first request; based on successfully verifying the credential: authenticating, by the first data daemon, the requestor using the secure authentication technique for the secure data source, establishing a connection with the secure data source for access to the secure data source by the application, and generating, by the first data daemon, a delegation token comprising a value based on an authentication secret of the secure data source; sending the delegation token to a second virtual compute node of the plurality of virtual compute nodes; sending, by a second data agent in the second virtual compute node, the delegation token to a second data daemon for a second request to access the secure data source intercepted by the second data agent; verifying, by the second data daemon, the delegation token from the second data agent; and ​ authenticate, based on verifying the delegation token, the security data source for the requestor that initiated the second request.

12. The method of claim 11, wherein the plurality of security data sources includes a second security data source that is protected by a second security authentication technique that is different from the security authentication technique used for the security data source.

13. The method of claim 11, wherein the delegation token is generated further based on a timestamp.

14. The method of claim 12, wherein the security authentication technique used for the security data source uses a first type of authentication secret, and the second security authentication technique used for the second security data source uses a second type of authentication secret that is different from the first type of authentication secret.

15. The method of claim 11, wherein the credential is a Kerberos credential and the credential issuing authority is a Kerberos Key Distribution Center (KDC).

16. A computing system comprising: a processor; and a non-transitory storage medium storing instructions executable on the processor to perform operations of: intercepting, by a first data agent in a first virtual compute node, a first request from an application in the first virtual compute node, the first request for access to a security data source for a requestor, the first virtual compute node being one of a plurality of virtual compute nodes of a compute cluster in a virtual data processing environment, the virtual data processing environment including a plurality of security data sources accessible to the plurality of virtual compute nodes, the security data source being protected by a security authentication technique; forwarding, by the first data agent, the first request to a first data daemon outside the first virtual compute node; verifying, by the first data daemon, a credential from a current application context associated with the first request with a credential issuing authority; and based on successfully verifying the credential: authenticating, by the first data daemon, the requestor using the security authentication technique for the security data source, establishing a connection with the security data source for access to the security data source by the application, and generating, by the first data daemon, a delegation token including a value based on an authentication secret of the security data source; sending the delegation token to a second virtual compute node of the plurality of virtual compute nodes; sending, by a second data agent in the second virtual compute node, the delegation token to a second data daemon for a second request to access the security data source intercepted by the second data agent; verifying, by the second data daemon, the delegation token from the second data agent; and based on verifying the delegation token, authenticating, by the second data daemon, the security data source for the requestor that initiated the second request. ​ ​ ​ 17. The computing system of claim 16, wherein the plurality of secure data sources includes a second secure data source that is protected by a second secure authentication technique that is different from the secure authentication technique used for the secure data source.

18. The computing system of claim 17, wherein the secure authentication technique used for the secure data source uses a first type of authentication secret and the second secure authentication technique used for the second secure data source uses a second type of authentication secret that is different from the first type of authentication secret.

19. The computing system of claim 16, wherein the value in the delegation token includes a signature, and wherein verifying, by the second data daemon, the delegation token comprises: computing, by the second data daemon, a signature based on the authentication secret of the secure data source, and comparing, by the second data daemon, the computed signature to the signature in the delegation token.

20. The computing system of claim 16, wherein the credential is a Kerberos credential and the credential issuer is a Kerberos Key Distribution Center (KDC). ​ ​

Images