Direct access to data lake files

Abstract

In example embodiments, a solution is provided that allows a user to directly interact with underlying object data lake storage in a very large scale service provider when accessing user data. The feature is referred to as direct access. By using the solution, the performance of the user system and the cloud system is improved without influencing functions and increasing complexity. As such, even for a Spark user, in Spark, the driver abstracts the communication path between the Spark application and the HDL file. This results in an optional solution that allows a user to reduce its cost by increasing complexity.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 703,570, entitled “DIRECT ACCESS OF DATA LAKE FILES”, filed on October 4, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application involves direct access to data lake files. Background Technology

[0004] A data lake is a single, centralized repository where organizations can store data in structured, unstructured, and semi-structured formats. This allows organizations to store, access, and analyze a wide variety of data in a single location more quickly and easily. Unlike databases, data stored in a data lake does not need to conform to a specific structured format. Instead, data can be stored in its raw or native format (typically as files or binary large objects (BLOBS)). Summary of the Invention

[0005] One aspect of this disclosure provides a system comprising: at least one hardware processor; a computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform an operation comprising: receiving from a user at a first virtual private cloud a request to access a file stored in a data lake object storage of a separate and different hyperscale service provider from the first virtual private cloud; generating a request to a data lake file application on a separate and different second virtual private cloud to authenticate the user; receiving a response from the data lake file application on the second virtual private cloud to grant the user access rights to the file; and, in response to receiving the response, directly accessing the data lake storage object from the first virtual private cloud to access the file.

[0006] In the example, this direct access uses the application programming interface provided in the data lake file client library on the first virtual private cloud.

[0007] In the example, the data lake file application determines the user's access permissions to the file based on trusted authorities listed in the cluster file container on the second virtual private cloud.

[0008] In the example, the request to access the file includes a special header indicating that the user is requesting direct access.

[0009] In the example, the response includes hyperscale service provider-specific data.

[0010] In an example, the hyperscaler-specific data includes a pre-signed uniform resource locator (URL).

[0011] In an example, the hyperscaler-specific data includes a temporary credential.

[0012] Another aspect of the disclosure provides a method comprising: receiving, at a first virtual private cloud, a request from a user to access a file stored in a data lake object store of a hyperscaler that is separate and distinct from the first virtual private cloud; generating a request to a data lake file application on a second virtual private cloud that is separate and distinct from the first virtual private cloud to authenticate the user; receiving a response from the data lake file application on the second virtual private cloud to grant the user access to the file; and responsive to receiving the response, directly accessing the data lake storage object from the first virtual private cloud to access the file.

[0013] In an example, the directly accessing the data lake storage object uses an application programming interface provided in a data lake file client library on the first virtual private cloud.

[0014] In an example, the data lake file application determines that the user should be granted access to the file based on trusted authorities listed in a cluster file container on the second virtual private cloud.

[0015] In an example, the request to access the file contains a special header that the user requests to be directly accessed.

[0016] In an example, the response includes hyperscaler-specific data.

[0017] In an example, the hyperscaler-specific data includes a pre-signed uniform resource locator (URL).

[0018] In an example, the hyperscaler-specific data includes a temporary credential.

[0019] Yet another aspect of the disclosure provides a non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving, at a first virtual private cloud, a request from a user to access a file stored in a data lake object store of a hyperscaler that is separate and distinct from the first virtual private cloud; generating a request to a data lake file application on a second virtual private cloud that is separate and distinct from the first virtual private cloud to authenticate the user; receiving a response from the data lake file application on the second virtual private cloud to grant the user access to the file; and responsive to receiving the response, directly accessing the data lake storage object from the first virtual private cloud to access the file.

[0020] In an example, the direct access data lake storage object uses an application programming interface provided in a data lake file client library on the first virtual private cloud.

[0021] In an example, the data lake file application determines that a user should be granted access to a file based on a trusted authority listed in a cluster file container on the second virtual private cloud.

[0022] In an example, the request to access the file contains a special header that the user requests direct access.

[0023] In an example, the response includes hyperscaler specific data.

[0024] In an example, the hyperscaler specific data includes a pre-signed uniform resource locator (URL). BRIEF DESCRIPTION OF DRAWINGS

[0025] The disclosure is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference numerals refer to similar elements.

[0026] Figure 1 is a block diagram illustrating a system for HDL file management, in accordance with example embodiments.

[0027] Figure 2 is a flow diagram illustrating a method for authorizing access to a file on a data lake, in accordance with example embodiments.

[0028] Figure 3 is a block diagram illustrating a software architecture, which can be installed on any one or more of the devices described above.

[0029] Figure 4 An illustration of a machine in the form of a computer system is shown, within which a set of instructions can be executed to cause the machine to perform any one or more of the methodologies discussed herein. DETAILED DESCRIPTION

[0030] The following description discusses illustrative systems, methods, techniques, instruction sequences, and computer machine program products. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide an understanding of various example embodiments of the present subject matter. It will be apparent, however, to one skilled in the art that

[0031] Files in a data lake can be stored in the data lake storage format. Data lake files can sometimes also be stored in in-memory data stores, such as SAP's HANA™ from Walldorf, Germany. The ability to store files in HANA in the data lake storage format is called HANA Data Lake (HDL) Files.

[0032] HDL files can be shared across multiple hyperscalers. Hyperscalers are companies or entities that typically operate at extremely large scales within the context of cloud computing and data center infrastructure. These organizations often manage vast amounts of data and computing resources across multiple geographic regions and have the ability to rapidly scale their operations to meet growing demands. Therefore, in systems with multiple hyperscalers (e.g., Amazon Web Services, Azure, etc.), files in each hyperscaler's data lake can be stored using HDL files. Thus, HDL files leverage object storage provided by the underlying hyperscaler to store all user data.

[0033] When accessing storage, users interact with the HDL file application and have no means of directly interacting with the underlying object storage. Users are unaware of the data's location in the background storage. The HDL file application is responsible for providing storage, and all access to the storage is performed through the Application Programming Interface (API) exposed by the HDL file.

[0034] This raises a technical issue. Since all communication between users and storage must go through HDL files, the total communication cost may be higher and the performance worse compared to direct communication.

[0035] Creating an abstraction layer comes with costs, so a balance needs to be struck between the level of abstraction and the perception of value.

[0036] In an example embodiment, a solution is provided that allows users to interact directly with the underlying object data lake storage in a hyperscale service provider when accessing user data. This feature is referred to as direct access. Using this solution improves the performance of both user systems and cloud systems without affecting functionality or adding complexity. This is also true for Spark users, where the driver abstracts the communication path between Spark applications and HDL files. This provides an alternative solution that allows users to reduce their costs by increasing complexity.

[0037] Apache Spark is an open-source distributed computing system designed for fast and efficient processing of large-scale datasets. It provides a unified analytics engine with built-in modules for various data processing tasks, such as:

[0038] 1. Batch Processing: Spark can handle large-scale data processing tasks, typically used for running big data jobs that require batch processing of large amounts of data.

[0039] 2. Streaming: It supports real-time data processing and analysis, allowing you to process streaming data as it arrives.

[0040] 3. SQL Queries: With Spark SQL, you can run SQL queries on your data, integrate with existing data sources, and enable various analytical operations.

[0041] 4. Machine Learning: Spark includes MLlib, a library of machine learning algorithms that facilitates scalable machine learning and data mining.

[0042] 5. Graph Processing: GraphX is Spark's API for graph processing, supporting analysis and manipulation of graph-structured data.

[0043] Spark is designed to be fast and scalable, utilizing in-memory computing to speed up data processing tasks. It can handle a wide range of workloads and is often used in conjunction with Hadoop's HDFS (Hadoop Distributed File System), but can also work with other data storage systems. Its ease of use and versatility make it a popular choice for big data applications and analysis.

[0044] Figure 1 is a block diagram illustrating a system 100 for HDL file management according to example embodiments. The system 100 includes a first virtual private cloud (VPC) 102 and a second VPC 104. Also present in the system are a plurality of object stores 106A, 106B, each operated by a different hyperscaler.

[0045] The HDL files are existing HANA Cloud services / components that are extended using the present solution. The HDL files component's responsibility in the HANA Cloud is to provide object storage using a hyperscaler platform-agnostic (hyperscaler platform-agnostic) Representational State Transfer (REST) API. Previously, the HDL files utilized the object storage 106A, 106B provided by the underlying hyperscaler to store all user data. In example embodiments, the HDL files now support an alternative mode of operation called "Direct Access". It will govern only the user's access to the storage, but the user data will be downloaded from the hyperscaler / directly uploaded to the hyperscaler. This design allows for improved performance (e.g., latency, throughput, bandwidth) and reduced cost. This is due to a reduction in compute cost on the HDL files side; a reduction in cost by avoiding the cost of transferring data across VPC boundaries.

[0046] Note that in this mode of operation, user authentication and authorization is still provided and enforced by the HDL files.

[0047] The first VPC 102 contains a Big Data Services (BDS) Kubernetes cluster 108. Kubernetes is a system for automating deployment, scaling, and management of containerized applications. Application containers are a virtualization method used by operating systems to deploy and run distributed applications without starting an entire virtual machine for each application.

[0048] Within the BDS Kubernetes cluster 108, a Spark application 110 runs a user workload that utilizes the HDL files as its storage. This is performed with the help of the HDL files Spark client libraries 112, which implement the Spark interface to talk to the HDL files service.

[0049] The HDL files Spark client libraries are libraries that allow integration between Spark and HDL files. These libraries use endpoints exposed by the HDL files to transfer data to the HDL files and do not require knowledge about the underlying object storage. This results in data transfer across VPCs, potentially incurring additional cost. Furthermore, the HDL files will need compute resources to process the data that is being transferred, incurring even more cost. In example embodiments, the HDL files Spark client libraries 112 now utilize the present solution, specifically Direct Access, to transfer data directly to the object storage, resulting in improved performance and reduced cost.

[0050] The HDL file application 114 runs within the HDL Kubernetes cluster 116 on the second VPC 104.

[0051] The hyperscale provider specific object stores 106A, 106B each ultimately hold the user's data content. Each hyperscale provider provides APIs to interact with the object store, which can typically be leveraged by the HDL file and are completely abstracted. However, when using direct access, the user interacts directly with the hyperscale provider using these APIs, which are not uniform across hyperscale providers. In this context, the HDL file application 114 is still responsible for dictating the transfer, and as such, will rely on the underlying hyperscale provider capabilities to do so. This includes generating pre-signed URLs, temporary credentials, or any other resources proven necessary, which will be analyzed and defined on a per-hyperscale provider basis.

[0052] It should be noted that the HDL file application 114 presents the user with credentials / resources with the minimum permissions to access the target file, so the hyperscale provider should have this capability as a prerequisite.

[0053] The existing API of the HDL file is unchanged, but a new optional HTTP header will be added to the OPEN and CREATE operations to allow the user to request data to be transferred directly from / to the underlying object store. The user will be able to request direct access as a hint, falling back to the default mode of operation if direct access is not available, and the user will be able to request direct access as a requirement, in which case the request fails if the requirement is not met.

[0054] When requesting direct access, no object content data transfer happens through the HDL file application 114. Instead, the HDL file application 114 will dictate the transfer by only returning a JSON response to the client, which contains the necessary information for the client to establish a connection with the hyperscale provider and transfer data directly.

[0055] The response from the HDL file will contain hyperscale provider specific data, and it is expected that the client will be able to interpret the response and act differently depending on the received hyperscale provider type. The returned data can contain pre-signed URLs, temporary credentials, or any other data proven to exist to allow the user to perform the operation.

[0056] For example, consider the OPEN call / test?op=OPEN, the user can request direct access by setting the header X-SAP-Accept-Direct-Access to "true". This header is interpreted by the HDL file as a hint, and if direct access is not enabled or not supported, the header will simply be ignored. If the user requires direct access, the header "X-SAP-Expect-Direct-Access: true" will be used, in which case the HDL file will respond with a failure message if direct access cannot be achieved.

[0057] The HDL file will return to the client the response header "X-SAP-Direct-Access: true", which indicates that direct access is being used to allow the client to interpret the response body correctly.

[0058] Consider that the underlying object store is "GCS" and that direct access is implemented, the response body of the HDL file will look like:

[0059]

[0060] Note that the response indicates the underlying storage type as well as hyperscaler-specific properties, so that the client can parse it and handle it correctly. One such property is the presence of header entries; any entries provided by the response must be used as headers in the request to the provided endpoint. In this example, the HDL file generates a pre-signed Uniform Resource Locator (URL) that allows the file to be read directly from the hyperscaler via HTTPS. Note that hyperscaler-specific information is also present in the URL itself, such as the bucket name and IAM user. This information will refer to resources provided only for the Cluster File Container (CFC) 118 to be accessed. Also, note that the URL is pre-signed with the key managed by the HDL file, so it is not possible for the user to change the path being accessed or the expiration time.

[0061] Furthermore, the OPEN call of the HDL file allows the user to specify a read range via offset and length query parameters. When direct access is used, these parameters will also be supported, however, given that most hyperscalers do not support these parameters, they will most likely not be implemented directly by the returned pre-signed URL. Instead, the offset and range will be injected by the user according to the request to the hyperscaler. Note that this is not a problem, since the HDL file does not support revoking authorization for the user to read a specific file part.

[0062] The CREATE operation will work similarly to the OPEN operation. However, it is important that the HDL file allow the user to take advantage of the hyperscale service provider upload capabilities available at the time and place, such as multipart upload or resumable upload. To do this, it can be necessary to return more information to the user than a simple pre-signed URL, such as temporary credentials.

[0063] Finally, the work of making the request to the specified hyperscale service provider will be delegated to the client / user, utilizing the information returned from the HDL file. The client will be provided with appropriate documentation that explains the format of the HDL file response, and how to interpret the data and correctly call the API. In the Apache Spark scenario, the work of consuming these APIs and interacting with the hyperscale service provider is delegated to our Spark library.

[0064] As for security, even though the data transfer will not go through the HDL file, the HDL file will still govern the transfer and enforce user authentication and authorization. Based on the trusted authorities configured in the cluster file container being accessed, as well as the roles and access policies assigned to the user, the user will only be able to request direct access if they have permission to do so. Once the user has performed the request and obtained a JSON response from the HDL file, the user is considered authorized, and anyone in possession of the JSON response will be able to interact with the hyperscale service provider.

[0065] The user will not have permission to operate on any other files outside of the target prefix. Files within the target prefix can also be used for read / write. In this context, special care must be taken with the HDL file, and direct access to prefixes that can violate the user-defined access policies is never allowed.

[0066] The HDL file will ultimately utilize the security concepts provided by the underlying hyperscale service provider to enforce communication between the user and the hyperscale service provider with minimal permissions. The hyperscale service provider can allow the creation of pre-signed URLs for specific files, as well as generate temporary credentials with restricted access, utilizing IAM users with permissions to access specific paths within a bucket.

[0067] Furthermore, the response returned by the HDL file will always grant temporary permissions to the user, and the permissions should always expire after a short time. This will also be enforced by utilizing the hyperscale service provider available capabilities, such as temporary pre-signed URLs and temporary credentials. If the user becomes un-authorized for a given file in the HDL file after they have direct access temporary permissions, the temporary permissions can remain valid until their expiration time, which is a reasonable trade-off given the eventually consistent nature of the system.

[0068] To minimize the risk of unauthorized personnel accessing a given CFC if they obtain temporary permission, some restrictions can be implemented. Object storage will be configured with a policy that will only allow access from private IP addresses, which will force users to perform access from within the same hyperscaler network and region. By forcing object storage itself to only be accessible from a certain IP range, resources such as pre-signed URLs and temporary credentials will also be automatically restricted. This concept can be extended, if needed, to allow users to specify specific IP ranges to allow.

[0069] As for user handling, new options will be added to the CREATE and OPEN operations to allow users to request direct access. When direct access is provided, the HDL file will return a well-documented response in JSON format that must be parsed and interpreted by the client. Clients that want to take advantage of direct access will need to implement specialized logic to interact with each hyperscaler of interest.

[0070] Thus, a mechanism is created in the control plane to allow applications to get authorized and express a desire for certain resources, and the control plane generates a URL to allow the application to go directly to the endpoint. This works across multiple hyperscalers because the correct URL can be obtained in a hyperscaler-agnostic way.

[0071] In some example embodiments, a JSON Web Token (JWT) can be included as a URL parameter to provide additional authentication capabilities. Also, in some instances, traceability of the use of the URL is important; e.g., when the URL was used and how often it can be tracked.

[0072] If there are multiple locations of a hyperscaler, the URL can point to the location closest to the user.

[0073] Also, the headers of the request can influence which URL is generated. For example, the request parameters can include a read range or specific columns. Even access to individual columns can be authorized independently.

[0074] Figure 2 is a flowchart showing a method 200 for authorizing access to files on a data lake, according to example embodiments. At operation 210, a request is received from a user at a first virtual private cloud to access a file stored in an object store of a data lake of a hyperscaler that is separate and distinct from the first virtual private cloud.

[0075] At operation 220, a request is generated to apply to a data lake file on a second virtual private cloud that is separate and distinct from the first virtual private cloud to authenticate the user.

[0076] At operation 230, a response is received from the data lake file application on the second virtual private cloud to grant the user access to the file.

[0077] At operation 240, responsive to receiving the response, the data lake storage object is directly accessed from the first virtual private cloud to access the file.

[0078] In view of the above implementation of the subject matter, the following list of examples is disclosed, wherein a feature of an individual example or a combination of features of the described examples, and optionally in combination with one or more features of one or more further examples, is also an additional example falling within the disclosure of the present application:

[0079] Example 1 is a system comprising: at least one hardware processor; a computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising: receiving, at a first virtual private cloud from a user, a request to access a file stored in a data lake object store of a hyperscaler that is separate and distinct from the first virtual private cloud; generating a request to a data lake file application on a second virtual private cloud that is separate and distinct from the first virtual private cloud to authenticate the user; receiving a response from the data lake file application on the second virtual private cloud to grant the user access to the file; and responsive to receiving the response, directly accessing the data lake storage object from the first virtual private cloud to access the file.

[0080] In Example 2, the subject matter of Example 1 includes, wherein the directly accessing the data lake storage object uses an application programming interface provided in a data lake file client library on the first virtual private cloud.

[0081] In Example 3, the subject matter of Example 2 includes, wherein the data lake file application determines that the user should be granted access to the file based on trusted authorities listed in a cluster file container on the second virtual private cloud.

[0082] In Example 4, the subject matter of Examples 1-3 includes, wherein the request to access the file contains a special header that the user requests to be directly accessed.

[0083] In Example 5, the subject matter of Examples 1-4 includes, wherein the response includes hyperscaler specific data.

[0084] In Example 6, the subject matter of Example 5 includes, wherein the hyperscaler specific data includes a pre-signed uniform resource locator (URL).

[0085] In Example 7, the subject matter of Examples 5-6 includes, wherein the hyperscaler specific data includes a temporary credential.

[0086] Example 8 is a method comprising: receiving, at a first virtual private cloud, a request from a user to access a file stored in a data lake object store of a hyperscaler that is separate and distinct from the first virtual private cloud; generating a request to a data lake file application on a second virtual private cloud that is separate and distinct from the first virtual private cloud to authenticate the user; receiving a response from the data lake file application on the second virtual private cloud to grant the user access to the file; and responsive to receiving the response, directly accessing the data lake storage object from the first virtual private cloud to access the file.

[0087] In Example 9, the subject matter of Example 8 includes, wherein the directly accessing the data lake storage object uses an application programming interface provided in a data lake file client library on the first virtual private cloud.

[0088] In Example 10, the subject matter of Example 9 includes, wherein the data lake file application determines that the user should be granted access to the file based on trusted authorities listed in a cluster file container on the second virtual private cloud.

[0089] In Example 11, the subject matter of Examples 8-10 includes, wherein the request to access the file contains a special header that the user requests to be directly accessed.

[0090] In Example 12, the subject matter of Examples 8-11 includes, wherein the response includes hyperscaler specific data.

[0091] In Example 13, the subject matter of Example 12 includes, wherein the hyperscaler specific data includes a pre-signed uniform resource locator (URL).

[0092] In Example 14, the subject matter of Examples 12-13 includes, wherein the hyperscaler specific data includes a temporary credential.

[0093] Example 15 is a non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving, at a first virtual private cloud, a request from a user to access a file stored in a data lake object store of a hyperscaler that is separate and distinct from the first virtual private cloud; generating a request to a data lake file application on a second virtual private cloud that is separate and distinct from the first virtual private cloud to authenticate the user; receiving a response from the data lake file application on the second virtual private cloud to grant the user access to the file; and responsive to receiving the response, directly accessing the data lake storage object from the first virtual private cloud to access the file.

[0094] In Example 16, the subject matter of Example 15 includes, wherein the direct access uses an application programming interface provided in a data lake file client library on the first virtual private cloud.

[0095] In Example 17, the subject matter of Example 16 includes, wherein the data lake file application determines that the user should be granted access to the file based on a trusted authority listed in a cluster file container on the second virtual private cloud.

[0096] In Example 18, the subject matter of Example 17 includes, wherein the request to access the file contains a special header for which the user is requesting direct access.

[0097] In Example 19, the subject matter of Examples 15-18 includes, wherein the response includes hyperscale service provider specific data.

[0098] In Example 20, the subject matter of Example 19 includes, wherein the hyperscale service provider specific data includes a pre-signed uniform resource locator (URL).

[0099] Example 21 is at least one machine readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement any of Examples 1-20.

[0100] Example 22 is an apparatus comprising means for implementing any of Examples 1-20.

[0101] Example 23 is a system for implementing any of Examples 1-20.

[0102] Example 24 is a method for implementing any of Examples 1-20.

[0103] Figure 3 is a block diagram 300 illustrating a software architecture 302, which can be installed on any one or more of the devices described above. Figure 3 is a non-limiting example of a software architecture, and that many other architectures can be implemented to facilitate the functionality described herein. In various embodiments, the software architecture 302 is implemented by a machine 400 Figure 4 of the machine 400, such as the processor 410, the memory 430, and the input / output (I / O) components 450. In this example architecture, the software architecture 302 can be conceptualized as a stack of layers, where each layer can provide a particular functionality. Figure 3 The software architecture 302 of FIG. 3 can be conceptualized as a stack of layers, where each layer can provide a particular functionality. For example, the software architecture 302 includes layers such as an operating system 304, libraries 306, frameworks 308, and applications 310. Operationally, the applications 310 invoke application programming interface (API) calls 312 through the software stack and receive messages 314 in response to the API calls 312.

[0104] In various implementations, the operating system 304 manages hardware resources and provides common services. The operating system 304 includes, for example, a kernel 320, services 322, and drivers 324. The kernel 320 acts as an abstraction layer between the hardware and the other software layers, consistent with some embodiments. For example, the kernel 320 provides memory management, processor management (for example, scheduling), component management, networking, and security settings, among other functionality. The services 322 can provide other common services that the other software layers can use. The drivers 324 are responsible for controlling or interfacing with the underlying hardware, according to some embodiments. For instance, the drivers 324 can include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers, flash memory drivers, serial communication drivers (for example, Universal Serial Bus [USB] drivers), Wi-Fi® drivers, audio drivers, power management drivers, and so forth.

[0105] In some embodiments, the libraries 306 provide a low-level common infrastructure used by the applications 310. The libraries 306 can include system libraries 330 (for example, C standard library) that can provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, the libraries 306 can include API libraries 332, such as media libraries (for example, libraries to support presentation and manipulation of various media formats such as Moving Picture Experts Group-4 [MPEG4], Advanced Video Coding [H.264 or AVC], Moving Picture Experts Group Layer-3 [MP3], Advanced Audio Coding [AAC], Adaptive Multi-Rate [AMR] audio codec, Joint Photographic Experts Group [JPEG or JPG], or Portable Network Graphics [PNG]), graphics libraries (for example, an OpenGL framework used to render two-dimensional [2D] and three-dimensional [3D] graphics in an application via commands to a display), database libraries (for example, SQLite to provide various relational database functions), web libraries (for example, WebKit to provide web browsing functionality), and the like. The libraries 306 also include a wide variety of other libraries 334 to provide many other APIs to the applications 310.

[0106] The framework 308 provides more abstracted infrastructure that can be utilized by the applications 310. For example, the framework 308 provides various graphical user interface (GUI) functions, higher-level resource management, higher-level location services, and so forth. The framework 308 can provide a broad spectrum of other APIs that can be utilized by the applications 310, some of which can be specific to a particular operating system 304 or platform.

[0107] In example embodiments, the applications 310 include a home application 350, a contacts application 352, a browser application 354, a book reader application 356, a location application 358, a media application 360, a messaging application 362, a game application 364, and a broad assortment of other applications such as a third party application 366. The applications 310 are programs that execute functions defined in the programs. Programs can be created using a variety of programming languages, such as object oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., C or assembly language) to name a few. In a specific example, the third party application 366 (e.g., an application developed using the ANDROID™ or IOS™ software development kit [SDK] by an entity other than the vendor of the particular platform) can be mobile software running on a mobile operating system such as IOS™, ANDROID™, WINDOWS® Phone, or another mobile operating system. In this example, the third party application 366 can invoke the API calls 312 provided by the operating system 304 to facilitate functionality described herein.

[0108] Figure 4 An illustration of a machine 400 in the form of a computer system is shown, within which a set of instructions, for causing the machine 400 to perform any one or more of the methodologies discussed herein, can be executed. Specifically, Figure 4 An illustration of a machine 400 in the form of a computer system is shown, within which a set of instructions, for causing the machine 400 to perform any one or more of the methodologies discussed herein, can be executed. Specifically, Figure 2 The machine 400 can further include a memory 404 (e.g., random access memory, read only memory), a storage device 406 (e.g., a disk, optical, tape), and a content Figures 1-2Instruction 416 transforms a general, unprogrammed machine 400 into a specific machine 400 programmed to perform the described and illustrated functions in the described manner. In alternative embodiments, machine 400 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 400 can operate as a server machine or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 400 can include, but is not limited to, server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web appliances, network routers, network switches, bridges, or any machine capable of sequentially or otherwise executing the actions specified by instruction 416 to be performed by machine 400. Furthermore, although only a single machine 400 is shown, the term "machine" should also be considered as a collection of machines 400 that individually or in combination execute instructions 416 to perform any one or more of the methods discussed herein.

[0109] Machine 400 may include processor 410, memory 430, and I / O components 450, which may be configured to communicate with each other, for example, via bus 402. In an example embodiment, processor 410 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processors 412 and 414 capable of executing instructions 416. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions 416 simultaneously. Although Figure 4 Multiple processors 410 are shown, but machine 400 may include a single processor 412 with a single core, a single processor 412 with multiple cores (e.g., a multi-core processor 412), multiple processors 412, 414 with a single core, multiple processors 412, 414 with multiple cores, or any combination thereof.

[0110] Memory 430 may include main memory 432, static memory 434, and memory cell 436, each of which may be accessed by processor 410 via bus 402. Main memory 432, static memory 434, and memory cell 436 store instructions 416 embodying any one or more of the methods or functions described herein. During execution of instructions 416 by machine 400, instructions 416 may also reside wholly or partially in main memory 432, static memory 434, memory cell 436, at least one of processor 410 (e.g., in the processor's cache memory), or any suitable combination thereof.

[0111] I / O component 450 may include a wide variety of components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O component 450 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that I / O component 450 may include components not included in... Figure 4 Many other components are shown. For the sake of simplicity in the following discussion, I / O components 450 are grouped according to function, and this grouping is not limiting. In various example embodiments, I / O components 450 may include output components 452 and input components 454. Output components 452 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. Input component 454 may include alphanumeric input components (e.g., a keyboard, a touchscreen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointing instrument), haptic input components (e.g., a physical button, a touchscreen that provides position and / or force for touch or touch gestures, or other haptic input components), audio input components (e.g., a microphone), etc.

[0112] In other example embodiments, I / O component 450 may include biometric component 456, motion component 458, environmental component 460 or position component 462, and a variety of other components. For example, biometric component 456 may include components for detecting expressions (e.g., hand gestures, facial expressions, vocal expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), and recognizing a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 458 may include accelerometer components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Environmental component 460 may include, for example, an illuminance sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting the concentration of hazardous gases or measuring pollutants in the atmosphere for safety purposes), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Location component 462 may include a positioning sensor component (e.g., a Global Positioning System (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), etc.

[0113] A wide variety of technologies can be used to implement communication. I / O component 450 may include communication component 464, which is operable to couple machine 400 to network 480 or device 470 via coupling 482 and coupling 472, respectively. For example, communication component 464 may include a network interface component or another suitable device that interfaces with network 480. In other examples, communication component 464 may include wired communication components, wireless communication components, cellular communication components, Near Field Communication (NFC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components for providing communication via other modes. Device 470 may be another machine or any of a wide variety of peripheral devices (e.g., coupled via USB).

[0114] Furthermore, communication component 464 may detect identifiers or include components operable to detect identifiers. For example, communication component 464 may include a radio-frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as QR codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying audio signals of the tag). Additionally, various information can be derived via communication component 464, such as location geolocation via Internet Protocol (IP), location triangulation via Wi-Fi® signals, location by detecting NFC beacon signals that can indicate a specific location, etc.

[0115] Various memories (i.e., the memories of 430, 432, 434 and / or the memory of processor 410) and / or storage units 436 may store one or more sets of instructions 416 and data structures (e.g., software) that embody or utilize any one or more of the methods or functions described herein. These instructions (e.g., instructions 416), when executed by processor 410, cause various operations to implement the disclosed embodiments.

[0116] As used herein, the terms “machine storage medium,” “device storage medium,” and “computer storage medium” refer to the same thing and are used interchangeably. These terms refer to one or more storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions and / or data. Therefore, these terms should be considered to include, but are not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), field-programmable gate arrays (FPGAs), and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage medium,” “computer storage medium,” and “device storage medium” specifically exclude carrier waves, modulated data signals, and other such media, at least some of which are covered under the term “signal medium” discussed below.

[0117] In various example embodiments, one or more portions of network 480 may be an ad hoc network, intranet, extranet, virtual private network (VPN), local-area network (LAN), wireless LAN (WWAN), wide-area network (WAN), wireless WAN (WWAN), metropolitan-area network (MAN), the Internet, a portion of the Internet, a portion of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, network 480 or a portion of network 480 may include a wireless or cellular network, and coupling 482 may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communication (GSM) connection, or another type of cellular or wireless coupling.In this example, Coupling 482 can implement any of a variety of data transmission technologies, such as Single Carrier Radio Transmission Technology (1xRTT), Evolution-Data Optimized (EVDO), General Packet Radio Service (GPRS), Enhanced Data Rates for GSM Evolution (EDGE), the third-generation partnership project (3GPP) including 3G, fourth-generation wireless (4G) wireless networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long-Term Evolution (LTE) standards, other standards defined by various standards-setting organizations, other long-range protocols, or other data transmission technologies.

[0118] Instructions 416 can be sent or received over network 480 via a transmission medium using a network interface device (e.g., a network interface component included in communication component 464) and utilizing any of several known transmission protocols (e.g., HTTP). Similarly, instructions 416 can be sent to or received from device 470 via a transmission medium using a coupling 472 (e.g., a peer-to-peer coupling). The terms "transmission medium" and "signal medium" refer to the same thing and can be used interchangeably in this disclosure. The terms "transmission medium" and "signal medium" should be considered to include any intangible medium capable of storing, encoding, or carrying instructions 416 for execution by machine 400, and include digital or analog communication signals or other intangible media to facilitate communication of such software. Therefore, the terms "transmission medium" and "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in a manner that encodes information in the signal.

[0119] The terms “machine-readable medium,” “computer-readable medium,” and “device-readable medium” refer to the same thing and may be used interchangeably in this disclosure. These terms are defined to include both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals.

Claims

1. A system comprising: At least one hardware processor; A computer-readable medium stores instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations, the operations including: At the first virtual private cloud, a user receives a request to access a file stored in a data lake object storage of a separate and different hyperscale service provider from the first virtual private cloud. Generate a request for a data lake file application on a second virtual private cloud that is separate from and different from the first virtual private cloud, to authenticate the user; Receive a response from the data lake file application on the second virtual private cloud to grant the user access to the file; and In response to receiving the response, the data lake storage object is accessed directly from the first virtual private cloud to access the file.

2. The system according to claim 1, wherein, The direct access uses the application programming interface provided in the data lake file client library on the first virtual private cloud.

3. The system according to claim 2, wherein, The data lake file application determines the user's access rights to the file based on trusted authorities listed in the cluster file container on the second virtual private cloud.

4. The system according to claim 1, wherein, The request to access the file includes a special header indicating that the user is requesting direct access.

5. The system according to claim 1, wherein, The response includes data specific to ultra-large-scale service providers.

6. The system according to claim 5, wherein, The hyperscale service provider-specific data includes pre-signed Uniform Resource Locators (URLs).

7. The system according to claim 5, wherein, The specific data from the ultra-large-scale service provider includes temporary credentials.

8. A method comprising: At the first virtual private cloud, a user receives a request to access a file stored in a data lake object storage of a separate and different hyperscale service provider from the first virtual private cloud. Generate a request for a data lake file application on a second virtual private cloud that is separate from and different from the first virtual private cloud, to authenticate the user; Receive a response from the data lake file application on the second virtual private cloud to grant the user access to the file; and In response to receiving the response, the data lake storage object is accessed directly from the first virtual private cloud to access the file.

9. The method according to claim 8, wherein, The direct access to the data lake storage object uses the application programming interface provided in the data lake file client library on the first virtual private cloud.

10. The method according to claim 9, wherein, The data lake file application determines the user's access rights to the file based on trusted authorities listed in the cluster file container on the second virtual private cloud.

11. The method according to claim 8, wherein, The request to access the file includes a special header indicating that the user is requesting direct access.

12. The method according to claim 8, wherein, The response includes data specific to ultra-large-scale service providers.

13. The method according to claim 12, wherein, The hyperscale service provider-specific data includes pre-signed Uniform Resource Locators (URLs).

14. The method according to claim 12, wherein, The specific data from the ultra-large-scale service provider includes temporary credentials.

15. A non-transitory machine-readable medium storing instructions, which, when executed by one or more processors, cause the one or more processors to perform an operation, the operation comprising: At the first virtual private cloud, a user receives a request to access a file stored in a data lake object storage of a separate and different hyperscale service provider from the first virtual private cloud. Generate a request for a data lake file application on a second virtual private cloud that is separate from and different from the first virtual private cloud, to authenticate the user; Receive a response from the data lake file application on the second virtual private cloud to grant the user access to the file; and In response to receiving the response, the data lake storage object is accessed directly from the first virtual private cloud to access the file.

16. The non-transitory machine-readable medium according to claim 15, wherein, The direct access to the data lake storage object uses the application programming interface provided in the data lake file client library on the first virtual private cloud.

17. The non-transitory machine-readable medium according to claim 16, wherein, The data lake file application determines the user's access rights to the file based on trusted authorities listed in the cluster file container on the second virtual private cloud.

18. The non-transitory machine-readable medium according to claim 17, wherein, The request to access the file includes a special header indicating that the user is requesting direct access.

19. The non-transitory machine-readable medium according to claim 15, wherein, The response includes data specific to ultra-large-scale service providers.

20. The non-transitory machine-readable medium according to claim 19, wherein, The hyperscale service provider-specific data includes pre-signed Uniform Resource Locators (URLs).

Images