Direct access to data lake files

Abstract

This solution provides a way for users to directly interact with the underlying object data lake storage in a hyperscaler when accessing user data. [Solution] This feature is called direct access. Using this solution, the performance of user systems and cloud systems is improved without affecting functionality or increasing complexity. This also applies to Spark users, where the driver abstracts the communication path between the Spark application and the HDL file. This results in an opt-in solution that allows users to reduce the costs associated with increased complexity.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of U.S. Provisional Application No. 63 / 703,570, filed on October 4, 2024, entitled "DIRECT ACCESS OF DATA LAKE FILES". U.S. Provisional Application No. 63 / 703,570 is hereby incorporated by reference in its entirety.

Background Art

[0002] A data lake is a single centralized repository where an organization can store data in structured, unstructured, and semi - structured formats. This enables an organization to store, access, and analyze a wide variety of data more quickly and easily in a single location. Different from a database, the data stored in a data lake does not need to conform to a specific structured format. Instead, the data can be stored in its raw or native format, usually as files or binary large objects (BLOBS).

Summary of the Invention

Means for Solving the Problems

[0003] This disclosure is illustrated by way of example and is not limited to the figures of the accompanying drawings. In the drawings, the same reference numerals indicate like elements.

Brief Description of the Drawings

[0004] [Figure 1] It is a block diagram showing a system for HDL file management according to an exemplary embodiment. [Figure 2] It is a flowchart showing a method for allowing access to files on a data lake according to an exemplary embodiment. [Figure 3] It is a block diagram showing a software architecture that can be installed on any one or more of the above - mentioned devices. [Figure 4] This diagram shows a schematic representation of a machine in the form of a computer system, in which a set of instructions can be executed to cause the machine to perform any one or more of the methods discussed herein. [Modes for carrying out the invention]

[0005] The following description discusses exemplary systems, methods, techniques, instruction sequences, and computer program products. For illustrative purposes, the following description includes numerous specific details to help understand the various exemplary embodiments of this subject matter. However, it will be apparent to those skilled in the art that various exemplary embodiments of this subject matter can be implemented without these specific details.

[0006] Files in a data lake can be stored in a data lake storage format. Data lake files can also, in some cases, be stored in in-memory datastores such as HANA™, commercially available from SAP in Waldorf, Germany. The ability to store files in HANA in a data lake storage format is referred to as HANA Data Lake (HDL) files.

[0007] HDL files can be shared among multiple hyperscalers. Hyperscalers are generally companies or entities that operate on an extremely large scale in the context of cloud computing and data center infrastructure. These organizations often manage vast amounts of data and computing resources across multiple geographical areas and have the ability to rapidly scale their operations and meet growing demands. Therefore, in a system with multiple hyperscalers (e.g., Amazon Web Services, Azure, etc.), files in the data lake of each hyperscaler may be stored using HDL files. Thus, HDL files utilize object storage provided by the underlying hyperscaler to store all user data.

[0008] When accessing storage, users interact with the HDL file application but have no means of directly interacting with the underlying object storage. Users are unaware of where their data is secretly stored. The HDL file application is involved in providing the storage, and all access to the storage is performed through the application program interface (API) exposed by the HDL file.

[0009] This presents technical challenges. Since all communication between the user and storage must go through HDL files, the overall communication cost can be higher in some cases, and performance may be degraded if communication were to occur directly.

[0010] Creating abstraction layers is costly, and therefore, a balance must be struck between the level of abstraction and the perceived value.

[0011] In an exemplary embodiment, a solution is provided that allows users to directly interact with the underlying object data lake storage in a hyperscaler when accessing user data. This feature is referred to as direct access. Using this solution, the performance of user systems and cloud systems is improved without impacting functionality or increasing complexity. This also applies to Spark users, where the driver abstracts the communication path between the Spark application and HDL files. This results in an opt-in solution that allows users to reduce the costs associated with increased complexity.

[0012] Apache Spark is an open-source distributed computing system designed to process large datasets quickly and efficiently. Apache Spark provides an integrated analytics engine with built-in modules for a variety of data processing tasks, including: 1. Batch Processing: Spark can handle large-scale data processing tasks, often used to run big data jobs that require processing large amounts of data in batches. 2. Stream Processing: Spark supports real-time data processing and analysis, enabling it to process streaming data as it arrives. 3. SQL Queries: Spark SQL allows you to execute SQL queries on data, integrate existing data sources, and enable various analytical operations. 4. Machine Learning: Spark includes MLlib, a library of machine learning algorithms that facilitates scalable machine learning and data mining. 5. Graph Processing: GraphX ​​is Spark's API for graph processing, enabling the analysis and manipulation of graph-structured data.

[0013] Spark is fast, scalable, and designed to accelerate data processing tasks using in-memory computing. It can handle a wide range of workloads and is often used in conjunction with Hadoop's HDFS (Hadoop Distributed File System), but can also work with other data storage systems. Because of its ease of use and versatility, Spark is a widely chosen choice for big data applications and analytics.

[0014] Figure 1 is a block diagram showing a system 100 for HDL file management according to an exemplary embodiment. System 100 includes a first virtual private cloud (VPC) 102 and a second VPC 104. The system also includes multiple object storages 106A and 106B, each of which is operated by a different hyperscaler.

[0015] HDL files are existing HANA cloud services / components that are extended using this solution. The HDL file component in the HANA cloud is responsible for providing object storage using the Hyperscaler Platform Independent Representational State Transfer (REST) ​​API. Traditionally, HDL files utilized object storage 106A and 106B provided by the base hyperscaler to store all user data. In exemplary embodiments, HDL files now support an alternative operating mode called "direct access." Direct access only controls user access to storage, while user data is downloaded / uploaded directly from / to the hyperscaler. This design allows for improved performance (e.g., latency, throughput, bandwidth) and reduced costs. This is due to reduced computing costs on the HDL file side, i.e., cost reductions resulting from avoiding data transfer across VPC boundaries.

[0016] Nevertheless, it should be noted that in this operating mode, user authentication and authorization are provided and enforced by the HDL file.

[0017] The first VPC, 102, contains the Big Data Services (BDS) Kubernetes cluster 108. Kubernetes is a system for automating the deployment, scaling, and management of containerized applications. Application containerization is a virtualization method used by operating systems to deploy and run distributed applications without launching an entire virtual machine for each application.

[0018] Within the BDS Kubernetes cluster 108, Spark application 110 runs a user workload that uses HDL files as storage for Spark application 110. This is done by the HDL file Spark client library 112, which implements a Spark interface to interact with the HDL file service.

[0019] The HDL file Spark client library is a library that enables integration between Spark and HDL files. These libraries transfer data to HDL files using endpoints exposed by the HDL file, eliminating the need for knowledge of the underlying object store. This can result in data being transferred across VPCs, potentially incurring additional costs. Furthermore, computing resources are required to process the data being transferred by the HDL file, adding further costs. In an exemplary embodiment, the HDL file Spark client library 112 here utilizes this solution, specifically direct access, to transfer data directly to the object store, improving performance and reducing costs.

[0020] The HDL file application 114 runs within the HDL Kubernetes cluster 116 on the second VPC 104.

[0021] The hyperscaler-specific object storages 106A, 106B each ultimately hold the user's data content. Each hyperscaler provides an API for interacting with the object storage, and the object storage can typically be utilized and fully abstracted by HDL files. However, when direct access is used, the user interacts directly with the hyperscaler using these APIs, and the APIs are not integrated with the hyperscaler. In this context, the HDL file application 114 still takes responsibility for controlling the transfer and thus depends on the basic hyperscaler functions to do so. This includes generating signed URLs, temporary authentication information, or other resources that have been proven necessary, and these resources are analyzed and defined per hyperscaler.

[0022] The HDL file application 114 presents the user with authentication information / resources that have the minimum permissions to access the target file, and thus it should be noted that the hyperscaler should have such functionality as a prerequisite.

[0023] The existing APIs of the HDL file are not changed, but new optional HTTP headers are added to the OPEN operation and the CREATE operation to enable the user to request to transfer data directly with the basic object storage. The user can request direct access as a hint and fallback to the default operating mode if direct access is not available, and can also request direct access as a requirement, in which case the request fails if direct access is not completed.

[0024] When direct access is requested, object content data transfer via the HDL file application 114 does not occur. Instead, the HDL file application 114 controls the transfer only by returning a JSON response to the client containing the information necessary for the client to establish a connection with the hyperscaler and transfer data directly.

[0025] The response from the HDL file will contain hyperscaler-specific data, and the client is expected to interpret the response and behave differently depending on the type of hyperscaler received. The returned data may include signed URLs, temporary credentials, or any other data that has been proven to be necessary to enable the user to perform an action.

[0026] For example, considering an OPEN call / test?op=OPEN, the user can request direct access by setting the header X-SAP-Accept-Direct-Access to "true". This header is interpreted as a hint by the HDL file, and if direct access is not enabled or supported, the header is simply ignored. If direct access is required by the user, the header "X-SAP-Expect-Direct-Access: true" is used, in which case the HDL file responds with a failure message if direct access cannot be completed.

[0027] The HDL file returns to the client a response header "X-SAP-Direct-Access: true" indicating that direct access is being used to enable the client to correctly interpret the response body.

[0028] Assuming the base object score is "GCS" and direct access is completed, the response body of the HDL file will be as follows: { "type": "GCS", "properties": { "headers": { "x-goog-custom-audit-hdlf-request-id":"r-BcrczRKLxlsG4InfQQJhYsiIk" }, "endpoint": "https: / / storage.googleapis.com / bucket / test?GoogleAccessId= <iam-uuid "method": "GET" } }

[0029] Note that the response will exhibit basic storage type and hyperscaler-specific characteristics so that the client can properly parse and deal with the response. One such characteristic is the presence of header entries, and whatever entries the response provides must be used as headers in requests to the provided endpoint. In this example, the HDL file generates a signed uniform resource locator (URL) that enables direct reading of files from the hyperscaler over HTTPS. Note that hyperscaler-specific information such as the bucket name and IAM user is also present in the URL itself. This information refers only to resources provisioned in the Cluster File Container (CFC) 118 being accessed. Also note that the URL is pre-signed with a key managed by the HDL file, and therefore it is impossible for the user to change the path or expiration time through which it is accessed.

[0030] Furthermore, the OPEN call to an HDL file allows the user to specify the read range via offset and length query parameters. These parameters are also supported when direct access is used, but are unlikely to be directly enforced by the returned signed URL, assuming most hyperscalers do not support this. Instead, the offset and range are inserted by the user when making the request to the hyperscaler. It should be noted that this is not an issue, assuming the HDL file does not support not allowing the user to read specific parts of the file.

[0031] The CREATE operation works similarly to the OPEN operation. However, if the HDL file is eligible for user hyperscalar upload features such as multipart upload or resumable upload, it is important to allow the user to fully utilize them. Therefore, it may be necessary to return more information to the user than just a signed URL, such as temporary authentication credentials.

[0032] Finally, the client / user is tasked with making a request to the specified hyperscaler containing the information returned from the HDL file. The client is given appropriate documentation describing the format of the HDL file response, as well as how to properly interpret the data and call the APIs. In the Apache Spark scenario, the job of consuming these APIs and interacting with the hyperscaler is delegated to the Spark library.

[0033] Regarding security, even if data transfer does not pass through the HDL file, the HDL file still controls the transfer and enforces user authentication and authorization. Users can only request direct access if they are authorized to do so based on the trusted authorities configured on the cluster file container being accessed, as well as the roles and access policies assigned to the user. Once a user makes a request and obtains a JSON response from the HDL file, the user is considered authorized, and anyone with the JSON response can interact with the hyperscaler.

[0034] Users do not have permission to manipulate any other files outside the target prefix. Files within the target prefix may also be available for reading / writing. In this context, particular care must be taken from HDL files to avoid enabling direct access to prefixes, which may violate user-defined access policies.

[0035] The HDL file ultimately uses the security concepts provided by the underlying hyperscaler to enable communication between the user and the hyperscaler with minimal permissions. The hyperscaler can then use an IAM user with permission to access specific paths within the bucket to create signed URLs targeting specific files, as well as generate temporary credentials with limited access. The techniques used are, precisely,

[0036] Furthermore, the response returned by the HDL file should always grant the user temporary permission, and this permission should always expire after a short period. This is also achieved by utilizing hyperscalar-enabled features such as temporary signed URLs and temporary credentials. If, after a user has obtained temporary permission for direct access, authorization to a given file in the HDL file is not obtained, the temporary permission remains valid until its expiration time, which is a reasonable compromise assuming the ultimate consistency of the system.

[0037] To minimize the risk of unauthorized personnel accessing a given CFC (even if they obtain temporary permission), several restrictions can be implemented. A policy can be set in object storage to allow access only from private IP addresses, ensuring that access is performed only by users within the same hyperscalar network and region. By making object storage itself accessible only from specific IP ranges, resources such as signed URLs and temporary credentials are also automatically restricted. If necessary, this concept can be extended to allow users to specify the specific IP ranges that are permitted.

[0038] Regarding user handling, new options are added to the CREATE and OPEN operations to allow users to request direct access. When direct access is provided, the HDL file returns a well-supported response in JSON format, which must be parsed and interpreted by the client. Clients wishing to utilize direct access must implement dedicated logic to interact with each target hyperscaler.

[0039] Therefore, a mechanism is created within the control plane to enable applications to be authorized and express their desires for specific resources, and the control plane generates URLs that allow applications to reach endpoints directly. This is valid across multiple hyperscalers because it allows obtaining the correct URL independently of the hyperscaler.

[0040] In some exemplary embodiments, a JSON Web Token (JWT) may be included as a URL parameter to provide additional authentication functionality. Additionally, in some cases, traceability of URL usage is important, such as being able to track when and how many times a URL was used.

[0041] If a hyperscaler has multiple locations, the URL can be directed to the location closest to the user.

[0042] Furthermore, request headers can also influence which URL is generated. For example, request parameters can include read ranges or specific columns. Access to individual columns may, in some cases, be authorized independently.

[0043] Figure 2 is a flowchart illustrating a method 200 for granting access to a file on a data lake, according to an exemplary embodiment. In operation 210, the first virtual private cloud receives a request from a user to access a file stored in a data lake object storage of a hyperscaler separate from the first virtual private cloud.

[0044] In operation 220, a request is generated to authenticate the user to a data lake file application on a second virtual private cloud, which is separate from and distinct from the first virtual private cloud.

[0045] In operation 230, a response is received from the data lake file application on the second virtual private cloud granting the user access to the file.

[0046] In operation 240, in response to receiving a response, the data lake storage object is accessed directly from the first virtual private cloud to access the file.

[0047] In view of the above-mentioned implementation forms of the subject matter, this application discloses the following list of embodiments, which may include only one feature of an embodiment, a combination of multiple features of the above embodiments, and, if applicable, a combination of one or more features of one or more further embodiments.

[0048] Embodiment 1 is a system comprising at least one hardware processor and a computer-readable medium storing instructions, wherein when an instruction is executed by at least one hardware processor, the system causes at least one hardware processor to perform an operation, the operation including receiving a request from a user to access a file stored in a hyperscaler's data lake object storage separate from the first virtual private cloud in a first virtual private cloud; generating a request to authenticate the user to a data lake file application on a second virtual private cloud separate from the first virtual private cloud; receiving a response from the data lake file application on the second virtual private cloud granting the user access rights to the file; and, in response to receiving the response, directly accessing the data lake storage object from the first virtual private cloud to access the file.

[0049] In Example 2, the subject matter of Example 1 is extended to include direct access to data lake storage objects using an application program interface provided in a client library of data lake files on a first virtual private cloud.

[0050] In Example 3, the subject of Example 2 is extended to include the data lake file application determining that the user should be granted access to files based on trusted authorities listed in a cluster file container on a second virtual private cloud.

[0051] In Example 4, the subject matter of Examples 1 to 3 is expanded to include a special header in which the request to access the file includes a dedicated header that directly requests access from the user.

[0052] In Example 5, the subject matter of Examples 1 to 4 is expanded to include the fact that the response includes hyperscaler-specific data.

[0053] In Example 6, the subject of Example 5 is expanded to include hyperscaler-specific data, which includes a signed uniform resource locator (URL).

[0054] In Example 7, the subject matter of Examples 5 and 6 is extended to include the fact that hyperscaler-specific data includes temporary authentication information.

[0055] Embodiment 8 is a method that includes the steps of: receiving a request from a user to access a file stored in a hyperscaler's data lake object storage separate from the first virtual private cloud in a first virtual private cloud; generating a request to authenticate the user to a data lake file application on a second virtual private cloud separate from the first virtual private cloud; receiving a response from the data lake file application on the second virtual private cloud granting the user access rights to the file; and, in response to receiving the response, directly accessing the data lake storage object from the first virtual private cloud to access the file.

[0056] In Example 9, the subject matter of Example 8 is extended to include the step of directly accessing data lake storage objects by using an application program interface provided in the client library of data lake files on the first virtual private cloud.

[0057] In Example 10, the subject of Example 9 is extended to include the data lake file application determining that the user should be granted access to files based on trusted authorities listed within a cluster file container on a second virtual private cloud.

[0058] In Example 11, the subject matter of Examples 8 to 10 is expanded to include a special header in which the request to access the file includes a dedicated header that directly requests access from the user.

[0059] In Example 12, the subject matter of Examples 8 to 11 is extended to include the fact that the response includes hyperscaler-specific data.

[0060] In Example 13, the subject of Example 12 is expanded to include hyperscaler-specific data, which includes a signed uniform resource locator (URL).

[0061] In Example 14, the subject matter of Examples 12 and 13 is expanded to include the fact that hyperscaler-specific data includes temporary authentication information.

[0062] Embodiment 15 is a non-temporary machine-readable medium storing instructions, wherein when the instructions are executed by one or more processors, the instructions cause one or more processors to perform an operation, the operation including receiving a request from a user to access a file stored in a hyperscaler's data lake object storage separate from the first virtual private cloud in the first virtual private cloud; generating a request to authenticate the user to a data lake file application on a second virtual private cloud separate from the first virtual private cloud; receiving a response from the data lake file application on the second virtual private cloud granting the user access rights to the file; and, in response to receiving the response, directly accessing the data lake storage object from the first virtual private cloud to access the file.

[0063] In Example 16, the subject matter of Example 15 is accessed directly using an application program interface provided in the client library of data lake files on the first virtual private cloud.

[0064] In Example 17, the subject of Example 16 is extended to include the data lake file application determining that the user should be granted access to files based on trusted authorities listed within a cluster file container on a second virtual private cloud.

[0065] In Example 18, the subject of Example 17 is expanded to include a request for accessing a file that includes a dedicated header requesting direct access from the user.

[0066] In Example 19, the subject matter of Examples 15 to 18 is expanded to include the fact that the response includes hyperscaler-specific data.

[0067] In Example 20, the subject of Example 19 is expanded to include hyperscaler-specific data, which includes a signed uniform resource locator (URL).

[0068] Example 21 is at least one machine-readable medium that, when executed by a processing circuit, contains instructions causing the processing circuit to perform an operation to carry out any of Examples 1 to 20.

[0069] Example 22 is an apparatus equipped with means for carrying out any of Examples 1 to 20.

[0070] Example 23 is a system for implementing any of Examples 1 to 20.

[0071] Example 24 is a method for carrying out any of Examples 1 to 20.

[0072] Figure 3 is a block diagram 300 showing the software architecture 302, which can be installed on any one or more of the devices described above. Figure 3 is only a non-limiting example of the software architecture, and it will be understood that numerous other architectures can be implemented to drive the functions described herein. In various embodiments, the software architecture 302 is implemented by hardware such as the machine 400 in Figure 4, which includes a processor 410, memory 430, and input / output (I / O) components 450. In this exemplary architecture, the software architecture 302 in Figure 3 can be conceptualized as a stack of layers, each layer which may provide a specific function. For example, the software architecture 302 includes layers such as an operating system 304, a library 306, a framework 308, and an application 310. Operationally, the application 310 invokes an application programming interface (API) call 312 through the software stack, according to several embodiments, and receives a message 314 in response to the API call 312.

[0073] In various implementations, the operating system 304 manages hardware resources and provides common services. The operating system 304 includes, for example, a kernel 320, services 322, and drivers 324. The kernel 320 acts as an abstraction layer between hardware and other software layers, according to several embodiments. For example, the kernel 320 performs, among several functions, memory management, processor management (e.g., scheduling), component management, network connectivity, and security settings. Services 322 can provide other common services for other software layers. Drivers 324 are involved in controlling or interfaceing with the underlying hardware. For example, drivers 324 may include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® low-energy drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus [USB] drivers), Wi-Fi® drivers, audio drivers, power management drivers, etc.

[0074] In some embodiments, library 306 provides a low-level common infrastructure used by application 310. Library 306 may include a system library 330 (for example, the C standard library), which may provide functions such as memory allocation functions, string manipulation functions, and mathematical functions. In addition, Library 306 may include API libraries 332, such as a media library (for example, libraries to support the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codecs, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (for example, the OpenGL framework used for rendering in 2D and 3D in a graphics context on a display), a database library (for example, SQLite for providing various relational database functions), and a web library (WebKit for providing web browsing functionality). Library 306 may also include a wide variety of other libraries 334 to provide numerous other APIs to application 310.

[0075] Framework 308 provides a high-level common infrastructure that can be utilized by applications 310. For example, Framework 308 provides various graphical user interface (GUI) functions, high-level resource management, high-level location services, and so on. Framework 308 can also provide a wide range of other APIs that can be utilized by applications 310, some of which may be applications specific to a particular operating system 304 or platform.

[0076] In exemplary embodiments, application 310 includes a wide range of other applications, such as a home application 350, a contacts application 352, a browser application 354, a book reader application 356, a location application 358, a media application 360, a messaging application 362, a game application 364, and a third-party application 366. Application 310 is a program that executes functions defined in the program. Using various programming languages, one or more of applications 310 can be created that are structured in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In specific examples, a third-party application 366 (for example, an application developed by an entity other than a particular platform vendor using the ANDROID® or IOS® Software Development Kit [SDK]) may be mobile software that runs on a mobile operating system such as IOS®, ANDROID®, WINDOWS® Phone, or another mobile operating system. In this example, the third-party application 366 can invoke API calls 312 provided by the operating system 304 to advance the functionality described herein.

[0077] Figure 4 shows a schematic diagram of machine 400 in the form of a computer system, within which a set of instructions may be executed to cause machine 400 to perform any one or more of the methods discussed herein. Specifically, Figure 4 shows a schematic representation of machine 400 in an exemplary form of a computer system, within which a set of instructions 416 (software, program, application, applet, app, or other executable code) may be executed to cause machine 400 to perform one or more of the methods discussed herein. For example, instruction 416 may cause machine 400 to perform the method of Figure 2. Additional or alternative instructions 416 may perform Figures 1, 2, etc. Instruction 416 transforms a general, unprogrammed machine 400 into a specific machine 400 programmed to perform the function described and illustrated. In alternative embodiments, machine 400 may operate as a standalone device or be coupled to other machines (e.g., networked). In a network-connected deployment, machine 400 may operate as a server or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 400 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a mobile phone, a smartphone, a mobile device, a wearable device (e.g., a smartwatch), a smart home device (e.g., a smart appliance), another smart device, a web appliance, a network router, a network switch, a network bridge, or any other machine capable of sequentially or otherwise executing instructions 416 that specify actions to be performed by machine 400.Furthermore, although only a single machine 400 is illustrated, the term “machine” shall be deemed to include a set of machines 400 that perform one or more of the methods discussed herein by performing instruction 416 individually or together.

[0078] Machine 400 may include a processor 410, memory 430, and I / O components 450, which may be configured to communicate with each other, for example, via a bus 402. In an exemplary embodiment, the processor 410 (for example, a central processing unit [CPU], a reduced instruction set computing [RISC] processor, a multiple instruction set computing [CISC] processor, a graphics processing unit [GPU], a digital signal processor [DSP], an application-specific integrated circuit [ASIC], a radio frequency integrated circuit [RFIC], another processor, or any suitable combination thereof) may include, for example, processors 412 and 414 capable of executing instruction 416. The term “processor” is intended to include multicore processors which may include two or more independent processors (sometimes referred to as “cores”) capable of executing instruction 416 simultaneously. Figure 4 shows multiple processors 410, but machine 400 may represent a single processor 412 with a single core, a single processor 412 with multiple cores (e.g., a multi-core processor 412), multiple processors 412, 414 with a single core, multiple processors 412, 414 with multiple cores, or any combination thereof.

[0079] Memory 430 may include main memory 432, static memory 434, and storage unit 436, each of which is accessible to the processor 410, for example, via bus 402. Main memory 432, static memory 434, and storage unit 436 store instructions 416 that embody any one or more of the methods and functions described herein. Instructions 416 may also reside entirely or partially in main memory 432, static memory 434, storage unit 436, processor 410 (for example, in the processor's cache memory), or any preferred combination thereof, while the instruction 416 is executed by machine 400.

[0080] The I / O component 450 may include a wide variety of components for receiving inputs, providing outputs, generating outputs, transmitting information, exchanging information, capturing measurements, and so on. The specific I / O component 450 included in a particular machine depends on the type of machine. For example, portable machines such as mobile phones are likely to include touch input devices or other such input mechanisms, while headless server machines are unlikely to include such touch input devices. It will be understood that the I / O component 450 may include numerous other components not shown in Figure 4. The I / O component 450 is grouped functionally only for the purpose of simplifying the following description, and the grouping is by no means limiting. In various exemplary embodiments, the I / O component 450 may include an output component 452 and an input component 454. Output components 452 may include visual components (e.g., displays such as plasma display panels [PDPs], light-emitting diode [LED] displays, liquid crystal displays [LCDs], projectors, or cathode ray tubes [CRTs]), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), and other signal generators. Input components 454 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photo-optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing devices), haptic input components (e.g., physical buttons, touchscreens that provide position and / or contact force or touch gestures, or other haptic input components), and voice input components (e.g., microphones).

[0081] In further exemplary embodiments, the I / O component 450 may include, among a variety of components, a biometric component 456, a motion component 458, an environmental component 460, or a location component 462. For example, the biometric component 456 may include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or electroencephalography), and identifying people (e.g., voice recognition, retinal recognition, face recognition, fingerprint recognition, or electroencephalogram-based recognition). The motion component 458 may include acceleration sensor components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), and the like. The environmental component 460 may include, for example, a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting concentrations of hazardous gases for safety or for measuring airborne pollutants), or other components that can provide indicators, measurements, or signals corresponding to the surrounding physical environment. The location component 462 may include, for example, a location information sensor component (e.g., a Global Positioning System [GPS] receiver component), an altitude sensor component (e.g., an altimeter, or a barometer for detecting atmospheric pressure from which altitude can be derived), a direction sensor component (e.g., a magnetometer), and the like.

[0082] Communication can be carried out using a wide variety of technologies. Each I / O component 450 may include a communication component 464 capable of operating to connect machine 400 to network 480 or device 470 via couplings 482 and 472, respectively. For example, communication component 464 may include a network interface component or another suitable device for interfacing with network 480. Further examples include communication component 464 including wired communication components, wireless communication components, cellular communication components, near-field communication (NFC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components for providing communication via other methods. Device 470 may be another machine or one of a wide variety of peripheral devices (e.g., connected via USB).

[0083] Furthermore, the communication component 464 may include components that detect identifiers or are capable of operating to detect identifiers. For example, the communication component 464 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor that detects one-dimensional barcodes such as Universal Product Code [UPC] barcodes, QR Code®, Aztec Code, DataMatrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). In addition, a wide variety of information can be derived through the communication component 464, such as location information via Internet Protocol (IP) geolocation, location information via Wi-Fi® signal triangulation, and location information by detecting NFC beacon signals that can indicate a specific location.

[0084] Various memories (i.e., 430, 432, 434, and / or the memory of processor 410) and / or storage unit 436 may store one or more sets of instructions 416 and data structures (e.g., software) that embody or are used by one or more of the methods or functions described herein. When these instructions (e.g., instructions 416) are executed by processor 410, they cause embodiments disclosed to be implemented in various operations.

[0085] In this specification, the terms “machine storage medium,” “device storage medium,” and “computer storage medium” mean the same thing and may be used interchangeably. This term refers to one or more storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions and / or data. Thus, this term is considered to include, but is not limited to, solid memory, including internal or external memory of a processor, as well as optical and magnetic media. Specific examples of machine storage medium, computer storage medium, and / or device storage medium include, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), field-programmable gate arrays (FPGAs), and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and non-volatile memory including CD-ROMs and DVD-ROM disks. "Machine storage media," "computer storage media," and "device storage media" specifically exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signaling media" as described below.

[0086] In various exemplary embodiments, one or more parts of network 480 could be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of a public switched telephone network (PSTN), a Plain Old Telephone Service (POTS) network, a mobile phone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, network 480 or a part of network 480 could include a wireless network or a cellular network, and coupling 482 could be a code division multiple access (CDMA) connection, a global system for mobile communication (GSM) connection, or another type of cellular or wireless coupling.In this example, Coupling 482 may implement any of the following diverse types of data transfer technologies: Single Carrier Radio Transmission Technology (1xRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data rates for GSM Evolution (EDGE) technology, third Generation Partnership Project (3GPP®) including 3G, fourth generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long-Term Evolution (LTE) standards, other standards defined by various standardization bodies, other long-range protocols, or other data transfer technologies.

[0087] Instruction 416 may be transmitted or received on network 480 using a transmission medium via a network interface device (e.g., a network interface component included in communication component 464) and utilizing one of several well-known transport protocols (e.g., HTTP). Similarly, Instruction 416 may be transmitted or received using a transmission medium via coupling 472 to device 470 (e.g., peer-to-peer coupling). The terms “transmission medium” and “signaling medium” are used interchangeably in this disclosure and mean the same thing. The terms “transmission medium” and “signaling medium” are to include any intangible medium on which Instruction 416 can be stored, encoded, or carried so that it can be executed by machine 400, and are to be considered to include digital or analog communication signals or other intangible medium to facilitate communication of such software. Thus, the terms “transmission medium” and “signaling medium” are to be considered to include any form of modulated data signal, carrier wave, etc. The term “modulated data signal” means a signal in which one or more of its characteristics are set or modified to encode information in the signal.

[0088] The terms “machine-readable medium,” “computer-readable medium,” and “device-readable medium” have the same meaning in this disclosure and may be used interchangeably. These terms are defined to include both machine storage media and transmission media. Thus, these terms include both storage devices / mediums and carrier waves / modulated data signals. [Explanation of Symbols]

[0089] 100 Systems 102. The first virtual private cloud (VPC) 104 Second VPC 106A, 106B Object Storage 108 Big Data Services (BDS) Kubernetes Cluster 110 Spark applications 112 HDL files Spark File Service 114 HDL File Applications 118 Cluster File Container (CFC) 302 Software Architecture 304 Operating Systems 306 Library 308 Frameworks 310 Applications 312 Application Programming Interface (API) Calls 314 Messages 320 kernels 322 Services 324 drivers 330 System Library 332 API Libraries 334 Other Libraries 350 Home Applications 352 Contacts Application 354 Browser applications 356 Book Reader Applications 358 Location-based applications 360 Media Applications 362 Messaging Applications 364 Game Applications 366 Third-Party Applications 400 Machines 410,412,414 processors 416 Command 417 PDF 430 memory 432 Main Memory 434 Static Memory 436 storage units 450 I / O components 452 output components 454 Input Components 456 Biometric Components 458 Exercise Components 460 Environmental Components 462 Position Component 464 Communication Components 470 devices 472 Coupling 480 Network 482 Coupling

Claims

1. At least one hardware processor, A computer-readable medium storing instructions, When the instruction is executed by the at least one hardware processor, it causes the at least one hardware processor to perform an operation, and the operation is In the first virtual private cloud, the system receives requests from users to access files stored in a hyperscaler's data lake object storage, which is separate from and distinct from the first virtual private cloud. To generate a request to authenticate the user to a data lake file application on a second virtual private cloud that is separate from and distinct from the first virtual private cloud, Receiving a response from the data lake file application on the second virtual private cloud granting the user access rights to the file, A system that, in response to receiving the aforementioned response, directly accesses the data lake storage object from the first virtual private cloud in order to access the file.

2. The system according to claim 1, wherein the direct access is performed using an application program interface provided in the client library of the data lake files on the first virtual private cloud.

3. The system according to claim 2, wherein the data lake file application determines that the user should be granted access to the file based on trusted authorities listed in the cluster file container on the second virtual private cloud.

4. The system according to claim 1, wherein the request for accessing the file includes a dedicated header for which the user directly requests access.

5. The system according to claim 1, wherein the response includes hyperscaler-specific data.

6. The system according to claim 5, wherein the hyperscaler-specific data includes a signed uniform resource locator (URL).

7. The system according to claim 5, wherein the hyperscaler-specific data includes temporary authentication information.

8. The first virtual private cloud receives a request from a user to access a file stored in a hyperscaler's data lake object storage, which is separate from and distinct from the first virtual private cloud. The steps include generating a request to authenticate the user to a data lake file application on a second virtual private cloud that is separate from and distinct from the first virtual private cloud, The steps include receiving a response from the data lake file application on the second virtual private cloud granting the user access rights to the file, A method comprising the step of directly accessing the data lake storage object from the first virtual private cloud in order to access the file, in response to receiving the aforementioned response.

9. The method according to claim 8, wherein the step of directly accessing the data lake storage object uses an application program interface provided in a client library of data lake files on the first virtual private cloud.

10. The method according to claim 9, wherein the data lake file application determines that the user should be granted access to the file based on trusted authorities listed in a cluster file container on the second virtual private cloud.

11. The method according to claim 8, wherein the request for accessing the file includes a dedicated header for which the user directly requests access.

12. The method according to claim 8, wherein the response includes hyperscaler-specific data.

13. The method according to claim 12, wherein the hyperscaler-specific data includes a signed uniform resource locator (URL).

14. The method according to claim 12, wherein the hyperscaler-specific data includes temporary authentication information.

15. A non-temporary machine-readable medium storing instructions, wherein, when an instruction is executed by one or more processors, it causes the one or more processors to perform an operation, and the operation is In the first virtual private cloud, the system receives requests from users to access files stored in a hyperscaler's data lake object storage, which is separate from and distinct from the first virtual private cloud. To generate a request to authenticate the user to a data lake file application on a second virtual private cloud that is separate from and distinct from the first virtual private cloud, Receiving a response from the data lake file application on the second virtual private cloud granting the user access rights to the file, Non-temporary machine-readable media, including, in response to receiving the aforementioned response, directly accessing the data lake storage object from the first virtual private cloud in order to access the file.

16. The non-temporary machine-readable medium according to claim 15, wherein direct access to the data lake storage object is performed using an application program interface provided in the client library of data lake files on the first virtual private cloud.

17. The non-temporary machine-readable medium according to claim 16, wherein the data lake file application determines that the user should be granted access to the file based on trusted authorities listed in a cluster file container on the second virtual private cloud.

18. The non-temporary machine-readable medium according to claim 17, wherein the request for accessing the file includes a dedicated header that the user directly requests access to.

19. The response is a non-temporary machine-readable medium according to claim 15, comprising hyperscaler-specific data.

20. The hyperscaler-specific data includes a signed uniform resource locator (URL) in the non-temporary machine-readable medium according to claim 19.

Images