Unified runtime directory

Abstract

Aspects of the present disclosure relate to a runtime directory for a cloud storage engine for unifying data lakes and data warehouses. The runtime directory may expose a single global domain of the cloud storage table through an endpoint of a query engine for a data lake and another endpoint of a query engine for a data warehouse. The runtime directory may allow the query engine of the data lake and the query engine of the data warehouse to query any cloud storage table by representing the data warehouse native table in a format compatible with the data lake and the data lake native table in a format compatible with the data warehouse.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation of U.S. patent application No. 18 / 201,243 filed on May 24, 2023, the disclosure of which is hereby incorporated by reference into this document. Background Art

[0003] A runtime catalog may correspond to a list of cloud storage tables for a data warehouse or data lake. The runtime catalog may include one or more properties describing the cloud storage tables that a query engine of the data lake or data warehouse uses to execute queries. A query engine of a data lake or data warehouse may have its own runtime catalog that cannot be used by other query engines. Since each runtime catalog is updated independently, this may lead to data silos and difficulties in synchronizing each runtime catalog. Summary of the invention

[0004] Aspects of the present disclosure relate to a runtime catalog for a cloud storage engine that unifies data lakes and data warehouses. The runtime catalog may expose a single universe of cloud storage tables through an endpoint for a query engine of a data lake and another endpoint for a query engine of a data warehouse. The runtime catalog may include a bidirectional translation layer to allow a query engine of a data lake and a query engine of a data warehouse to query any cloud storage table, regardless of whether the format of the cloud storage table is compatible with the data lake or the data warehouse. The translation layer may represent data warehouse native tables in a format compatible with the data lake, and may represent data lake native tables in a format compatible with the data warehouse.

[0005] Aspects of the present disclosure provide a method for processing queries, including: receiving, by one or more processors, a request from a query engine to access data from a cloud storage table residing in a cloud storage system; determining, by the one or more processors, a type of table metadata required for the query engine to access data from the cloud storage table; generating, by the one or more processors, table metadata compatible with the query engine using a mapping between formats of different cloud storage systems; and providing, by the one or more processors, the table metadata to the query engine.

[0006] In an example, the method further includes: receiving, by one or more processors, a request to generate a cloud storage table from a second query engine; and generating, by one or more processors, a cloud storage table in a format compatible with the second query engine; wherein the query engine and the second query engine have incompatible formats. In another example, determining the type of table metadata further includes identifying a format of the query engine, wherein the query engine is formatted for a data lake and the cloud storage table is formatted for a data warehouse, or the query engine is formatted for a data warehouse and the cloud storage table is formatted for a data lake. In yet another example, the mapping includes a mapping of compatible features between a format of the query engine and a format of the cloud storage table.

[0007] In yet another example, the method further includes determining, by one or more processors, that the cloud storage system is associated with one or more governance policies. In yet another example, the method further includes: determining, by one or more processors, that the query engine supports one or more governance policies; and retrieving data directly from the cloud storage table by one or more processors. In yet another example, the method further includes: determining, by one or more processors, that the query engine does not support one or more governance policies; and retrieving data from the cloud storage table by one or more processors through a unified runtime application programming interface (API). In yet another example, the method further includes: determining, by one or more processors, that updates to the cloud storage table do not affect one or more governance policies; and verifying updates to the cloud storage table by one or more processors. In yet another example, the method further includes: determining, by one or more processors, that updates to the cloud storage table affect one or more governance policies; and rejecting updates to the cloud storage table by one or more processors.

[0008] In yet another example, the method further includes storing, by one or more processors, file-level metadata in a metadata management system. In yet another example, the method further includes storing, by one or more processors, lineage of data from a cloud storage table. In yet another example, the method further includes supporting, by one or more processors, unstructured data assets via an object table.

[0009] Another aspect of the present disclosure provides a system including: one or more processors; and one or more storage devices, which are coupled to the one or more processors and store instructions, which when executed by the one or more processors cause the one or more processors to perform operations for processing queries, the operations including: receiving a request from a query engine to access data from a cloud storage table residing in a cloud storage system; determining a type of table metadata required for the query engine to access the data from the cloud storage table; using a mapping between formats of different cloud storage systems to generate table metadata compatible with the query engine; and providing the table metadata to the query engine.

[0010] In an example, the operation further includes: receiving a request from a second query engine to generate a cloud storage table; and generating the cloud storage table in a format compatible with the second query engine; wherein the query engine and the second query engine have incompatible formats. In another example, determining the type of table metadata further includes identifying a format of the query engine, wherein the query engine is formatted for a data lake and the cloud storage table is formatted for a data warehouse, or the query engine is formatted for a data warehouse and the cloud storage table is formatted for a data lake. In yet another example, the mapping includes a mapping of compatible features between a format of the query engine and a format of the cloud storage table.

[0011] In yet another example, the operation also includes determining that the cloud storage system is associated with one or more governance policies. In yet another example, the operation also includes: determining that the query engine supports the one or more governance policies and directly retrieving data from the cloud storage table; or determining that the query engine does not support the one or more governance policies and retrieving data from the cloud storage table through a unified runtime application programming interface (API). In yet another example, the operation also includes: determining that an update to the cloud storage table does not affect the one or more governance policies and verifying the update to the cloud storage table; or determining that the update to the cloud storage table affects the one or more governance policies and rejecting the update to the cloud storage table.

[0012] Yet another aspect of the present disclosure provides a non-transitory computer-readable medium for storing instructions, which when executed by one or more processors causes the one or more processors to perform operations for processing a query, the operations including: receiving a request from a query engine to access data from a cloud storage table residing in a cloud storage system; determining a type of table metadata required for the query engine to access the data from the cloud storage table; generating table metadata compatible with the query engine using a mapping between formats of different cloud storage systems; and providing the table metadata to the query engine. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Depicted is a block diagram of an example unified runtime catalog for a cloud storage system that unifies data lakes and data warehouses in accordance with aspects of the present disclosure.

[0014] Figure 2 Depicted is a block diagram of an example unified runtime catalog in accordance with aspects of the present disclosure.

[0015] Figure 3 Depicted is a block diagram of an example computing environment implementing a unified runtime directory for a cloud storage system in accordance with aspects of the present disclosure.

[0016] Figure 4Depicted is a flow diagram of an example process for processing a query in accordance with aspects of the present disclosure.

[0017] Figure 5 Depicted is a flow diagram of an example process for directing query processing when associated with a governance policy in accordance with aspects of the present disclosure.

[0018] Figure 6 Depicted is a flow diagram of an example process for managing updates to cloud storage tables when associated with a governance policy in accordance with aspects of the present disclosure. DETAILED DESCRIPTION

[0019] The present technology generally relates to a runtime catalog for a cloud storage engine that unifies data lakes and data warehouses. A data lake is a repository that stores large amounts of data in its natural, raw, or native format (e.g., objects, blobs, or files). The data can be structured, semi-structured, or unstructured. The runtime catalog can expose a single universe of tables through an endpoint for a query engine for the data lake and another endpoint for a query engine for the data warehouse.

[0020] The runtime catalog may correspond to a list of cloud storage tables including properties of the cloud storage tables, such as name, mode, access control list (ACL), partitioning and clustering schemes, references to storage containing the cloud storage tables, and / or other runtime metadata / configuration. Query engines of data lakes and data warehouses may use the properties of the tables in the runtime catalog to perform queries. A data warehouse may have its own embedded runtime catalog, while a data lake may have a standardized runtime catalog to which various open source engines may connect.

[0021] The query engines of data lakes and data warehouses typically have their own runtime catalogs that are typically not usable by other query engines, creating data silos. Data silos simply mean that such data is typically not easily accessible or shared by other query engines. Keeping the runtime catalog in sync can be difficult because the tools or engines operating on the catalog can only update their own catalogs. When the runtime catalog deviates, many problems can occur. For example, table attributes may be missing, duplicated, and / or outdated, making it difficult to find data. In addition, if table attributes are repeated, implementing a specific security policy on one table may still inadvertently expose another table. As another example, the preferred query engine may not work with the runtime catalog in which the table is defined. As yet another example, advanced catalog features cannot be used because standardization occurs on the lowest common denominator of the runtime catalog. If two catalogs support incompatible types of fine-grained access control (FGAC), FGAC cannot be used.

[0022] A unified runtime catalog for data lakes and data warehouses can address these issues. A unified runtime catalog can support multiple table formats, including warehouse native tables as well as open source formats. For example, a unified runtime catalog can support tabular tables as well as semi-structured and / or unstructured data through object tables. Since data lakes and data warehouses have incompatible representations of tables and their attributes, the unified runtime catalog contains a bidirectional translation layer to allow query engines to query each other's tables. In this sense, the unified runtime catalog unifies data from different sources in the sense that the data can be accessed from different query engines corresponding to each different data source.

[0023] The translation layer can represent data warehouse native tables in a format compatible with the data lake, and can represent data lake native tables in a format compatible with the data warehouse. The translation layer can use mappings of compatible features of the runtime catalogs of the data lake and the data warehouse. The translation layer can use mappings to represent table properties between data lake compatible formats and data warehouse compatible formats. As examples of compatible features, the translation layer can represent schemas, advanced data types, column identifiers, special characters in column or table names, table properties that control data layout, sorting rules for controlling how data is sorted, ACLs, and / or FGAC. Since open source query engines may have different support for FGAC, the translation layer can also redirect open source query engines through the data lake runtime API when necessary to provide support for FGAC.

[0024] The unified runtime catalog can create, list and / or manage databases, tables and their properties through both the data lake endpoint and the data warehouse endpoint and through the data definition language (DDL) received from the query engine of the data lake or data warehouse. The unified runtime catalog can further return the table definition with metadata to the query engine of the data lake or data warehouse to process the query. The table definition allows the query engine to read and write the table.

[0025] For example, the unified runtime directory may receive a request from a first query engine to create a table. The unified runtime directory may generate a table in a format compatible with the first query engine. The unified runtime directory may receive a request from a second query engine to access data from the table (e.g., read and / or write data from the table). The unified runtime directory may determine that the request is from the second query engine, which may be incompatible with the format of the table. For example, the first query engine may be used for a data lake and the second query engine may be used for a data warehouse, or the first query engine may be used for a data warehouse and the second query engine may be used for a data lake. The translation layer of the unified runtime directory may use a mapping to represent the data in a format compatible with the second query engine. The unified runtime directory may return the data to the second query engine so that the second query engine can process the query.

[0026] For table definitions that are expensive to generate due to the need to filter millions of files to return files relevant to a query, the unified runtime catalog can use a metadata management system to store file-level metadata as a cache or primary source of truth to accelerate these operations. The metadata management system can include data about the state of the table for preliminary pruning of files, as well as table and column statistics that enable the query engine to make better decisions to process queries, such as using broadcast joins or performing dynamic predicate pruning using table schemas.

[0027] For some table formats, the unified runtime catalog can also return alternative table definitions that represent different ways to read data. For example, some query engines can read a table by getting the data file directly from the data lake, or read the same table through the data lake runtime API. Use the data lake runtime API to enable FGAC for query engines that do not support FGAC. The unified runtime catalog can select a table definition based on whether the table has FGAC established and whether the query engine supports FGAC. The unified runtime catalog ensures a uniform implementation of FGAC regardless of whether the query is processed on the data warehouse side or the data lake side.

[0028] The unified runtime catalog may further create and / or manage ACLs and FGACs. Example ACLs and / or FGACs may include column security, data masking, and / or row filtering. The unified runtime catalog may ensure that the query engine does not commit changes that may affect the ACLs and / or FGACs. For example, a query engine of a data lake that is unaware of the row access policy may attempt to rename a column that is mentioned in the row access policy. Doing so would violate the policy and inadvertently expose data. The unified runtime catalog may validate all table updates and reject updates that affect the ACLs and / or FGACs.

[0029] The unified runtime catalog can also store common data lineage for data lakes and data warehouses. The unified runtime catalog can provide ancestors and descendants for any table, regardless of which query engine performed the operation.

[0030] The unified runtime catalog may support unstructured data assets through an object table. The object table may correspond to a structured query language (SQL) interface for object storage metadata. The object table may include a list of objects with names, sizes, hashes, etc. The object table may allow files with arbitrary content to be represented in SQL.

[0031] Figure 1A block diagram of an example unified runtime catalog 100 for a cloud storage system that unifies data lakes and data warehouses is depicted. The unified runtime catalog 100 may include a data lake window 102 and a data warehouse window 104 to expose table entries 106 and governance policies 108 to a data lake query engine 110 and a data warehouse query engine 112, respectively. The example data lake query engine 110 may include an open source engine. The example data warehouse query engine 112 may include a warehouse native API.

[0032] The cloud storage system may include a data lake storage 114 and a data warehouse storage 116. The data lake storage 114 (such as an object store) may correspond to a repository containing raw, unstructured data, while the data warehouse storage 116 may correspond to a repository containing structured data. Based on metadata from the table entries 106 and / or the governance policies 108, the data lake query engine 110 may access data directly from the data lake storage 114, or may access data from the data lake storage 114 or the data warehouse storage 116 through a unified runtime 118. Similarly, based on metadata from the table entries 106 and / or the governance policies 108, the data warehouse query engine 112 may access data from the data lake storage 114 or the data warehouse storage 116 through a unified runtime 118. The unified runtime 118 may present the query engine 112 with a unified API to read data, regardless of whether the data is stored in the data lake storage 114 or the data warehouse storage 116. The unified runtime 118 can abstract whether the data files are stored in the data lake storage 114 or the data warehouse storage 116. The unified runtime 118 can also implement FGAC. The unified runtime 118 can further speed up query processing by using metadata caching to prune data files and reduce the amount of data processed by the query engine 112.

[0033] The unified runtime catalog 100 may correspond to a list of cloud storage tables of the data lake storage 114 and the data warehouse storage 116. The unified runtime catalog 100 may include table entries 106 and / or governance policies 108. The table entries 106 may correspond to metadata and / or configurations of cloud storage tables. Example table entries 106 may include patterns like partitioning and clustering patterns, table attributes like names or other identifiers, or pointers to governance policies like ACLs. The table entries 106 may also include common data lineage for providing ancestors or derivatives of cloud storage tables, regardless of whether executed by the data lake query engine 110 or the data warehouse query engine 112. The governance policies 108 may correspond to data security for accessing cloud storage tables. Example governance policies 108 may include ACLs, column security, data masking, row filtering, k-anonymity, or differential privacy. The data lake query engine 110 and the data warehouse query engine 112 may access the metadata of the table entries 106 and / or governance policies 108 to perform queries.

[0034] The unified runtime catalog 100 may include a metadata management system 120 to store file-level metadata to speed up the generation of table metadata to provide to the data lake query engine 110 or the data warehouse query engine 112. For example, generating some table definitions may be computationally expensive, requiring a lot of processing power to filter millions of files to return metadata relevant to the query. For example, generating metadata for queries associated with Hive partitioned tables may be computationally expensive. The metadata management system 120 may store file-level metadata as a cache or primary source of truth to preliminarily prune metadata so that the unified runtime catalog does not have to search as many files. In addition, the metadata management system 120 may store file-level metadata to enable the query engine to improve query processing. For example, the metadata management system 120 may allow the query engine to perform dynamic predicate pruning using broadcast joins or table patterns.

[0035] The unified runtime directory 100 can avoid the difficulties that arise when maintaining separate runtime directories for the data lake and the data warehouse. For example, the unified runtime directory 100 can make it easier to find data because the table entries 106 can be consistent for the data lake query engine 110 and the data warehouse query engine 112. The unified runtime directory 100 can avoid missing, duplicated, and / or outdated table entries that can occur in separate runtime directories that have diverged. As another example, the unified runtime directory 100 can make it easier to share data because users with different engines can share the same table definitions and the same ACLs without involving cumbersome and error-prone export / load cycles into engine-specific directories. As yet another example, the unified runtime directory 100 can improve security through common governance policies 106 for the data lake query engine 110 and the data warehouse query engine 112. The unified runtime directory 100 can avoid the lowest common denominator security standardization that can occur in separate runtime directories, where some governance policies cannot be used if the two runtime directories support incompatible types of governance policies. The unified runtime catalog 100 may also prevent inadvertent exposure of data by enforcing governance policies on one duplicate cloud storage table but not another duplicate cloud storage table.

[0036] Figure 2 A block diagram of an example unified runtime directory 200 is depicted. The unified runtime directory 200 may correspond to Figure 1 . The unified runtime catalog 200 may include a data lake endpoint 202 and a data warehouse endpoint 204 to interface with a data lake query engine and a data warehouse query engine, respectively. The data lake endpoint 202 and the data warehouse endpoint 204 may correspond to an API or any other interface to expose metadata to the data lake query engine and the data warehouse query engine.

[0037] The unified runtime catalog 200 may include a cloud storage table inventory 206 with table metadata 208 in any table format, including data warehouse native table formats and data lake native table formats. For example, the unified runtime catalog 200 may support table tables and support semi-structured or unstructured data through object tables. Example semi-structured or unstructured data may include images, videos, documents, and / or any files with non-table content. The object table may correspond to a structured query language (SQL) interface of the object storage metadata. The object table may include a list of objects, including their names, sizes, hashes, etc., to allow files with arbitrary content to be represented in SQL. The unified runtime catalog 200 may also include one or more governance policies 210 to provide data security.

[0038] To address the incompatible representations of tables in data lakes and data warehouses, the unified runtime catalog 200 includes a bidirectional translation layer 212 to allow data lake query engines and data warehouse query engines to process queries regardless of table formats. The translation layer 212 can represent cloud storage table manifests 206, including table metadata 208, and / or governance policies 210 in a format compatible with the data lake or data warehouse. For example, the translation layer 212 can represent data warehouse native tables in a format compatible with the data lake, and can represent data lake native tables in a format compatible with the data warehouse. The translation layer 212 can also redirect data lake query engines that do not support FGAC through a unified runtime (such as a data lake runtime API) to provide support for FGAC.

[0039] The translation layer 212 may include a mapping 214 of compatible features of the unified runtime catalog 200 for the data lake and the data warehouse. Examples of compatible features include schemas, advanced data types, column identifiers, special characters in column or table names, table properties that control data layout, ACLs, and / or FGACs. Example advanced data types may include STRUCT, ARRAY, BIGNUMERIC, JSON, and / or GEOGRAPHY. Example table properties that control data layout may include partition configuration and / or cluster configuration. Example FGACs may include column security, row filtering, data masking, and / or k-anonymity or differential privacy.

[0040] Figure 3 A block diagram of an example computing environment 300 implementing a unified runtime directory 302 for a cloud storage system is depicted. The unified runtime directory 302 may correspond to Figure 1 The unified runtime directory 100 or Figure 2 302. The unified runtime directory 302 may be implemented on one or more devices having one or more processors located in one or more locations, such as in a server computing device 304. The client computing devices 306 and the server computing devices 304 may be communicatively coupled to one or more storage devices 308 via a network 310. The server computing devices 304 and the storage devices 308 may form part of a cloud computing system 312 for cloud computing services, such as infrastructure as a service (IaaS), platform as a service (PaaS), and / or software as a service (SaaS).

[0041] For example, client computing device 306 may use cloud computing system 312 as a service to provide software applications such as accounting, word processing, inventory tracking, fraud detection, file sharing, video sharing, audio sharing, communications, or gaming. As another example, client computing device 306 may access cloud computing system 312 as part of one or more operations that employ machine learning, deep learning, and / or artificial intelligence techniques to train a software application. Cloud computing system 312 may provide model parameters that may be used to update a machine learning model for a software application.

[0042] The storage device 308 may be a combination of volatile and non-volatile memory and may be located at the same or a different physical location as the computing devices 304, 306. For example, the storage device 308 may include any type of non-transitory computer-readable medium capable of storing information, such as a hard disk, a solid-state drive, a tape drive, an optical storage device, a memory card, ROM, RAM, DVD, CD-ROM, a writable memory, and a read-only memory.

[0043] The server computing device 304 may include one or more processors 314 and memory 316. The memory 316 may store information accessible by the processor 314, including instructions 318 executable by the processor 314. The memory 316 may also include data 320 that may be retrieved, manipulated, or stored by the processor 314. The memory 316 may be a type of non-transitory computer-readable medium capable of storing information accessible by the processor 314, such as volatile memory and non-volatile memory. The processor 314 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0044] Instructions 318 may include one or more instructions that, when executed by processor 314, cause one or more processors to perform actions defined by instructions 318. Instructions 318 may be stored in an object code format for direct processing by processor 314 or in other formats including a collection of interpretable scripts or independent source code modules that may be interpreted on demand or pre-compiled. Instructions 318 may include instructions for implementing unified runtime directory 302. Unified runtime directory 302 may be executed using processor 314 and / or using other processors located remotely from server computing device 304.

[0045] Data 320 may be retrieved, stored, or modified by processor 314 according to instructions 318. Data 320 may be stored in a computer register, in a relational or non-relational database as a table with a plurality of different fields and records, or as a JSON, YAML, proto, or XML document. Data 320 may also be formatted in a computer-readable format such as, but not limited to, binary values, ASCII, or Unicode. In addition, data 320 may include information sufficient to identify the relevant information, such as a number, descriptive text, proprietary code, pointer, reference to data stored in other storage (including other network locations), or information used by a function to calculate the relevant data.

[0046] The client computing device 306 may also be configured similarly to the server computing device 304, with one or more processors 322, memory 324, instructions 326, and data 328. The client computing device 306 may also include a client input 330 and a client output 332. The client input 330 may include any suitable mechanism or technology for receiving input from a client, such as a keyboard, mouse, mechanical actuator, soft actuator, touch screen, microphone, and sensor.

[0047] The server computing device 304 may be configured to transmit data to the client computing device 306, and the client computing device 306 may be configured to display at least a portion of the received data on a display implemented as part of the client output 332. The client output 332 may also be used to display an interface between the client computing device 306 and the server computing device 304. The client output 332 may alternatively or additionally include one or more speakers, transducers or other audio outputs, tactile interfaces, or other tactile feedback that provides non-visual and non-audible information to a client of the client computing device 306.

[0048] although Figure 3Processors 314, 322 and memories 316, 324 are shown as being located within computing devices 304, 306, but the components described herein (including processors 314, 322 and memories 316, 324) may include multiple processors and memories that can operate at different physical locations rather than within the same computing device. For example, some of data 318, 326 and instructions 320, 328 may be stored on a removable SD card, while other instructions are stored in a read-only computer chip. Some or all of data 31, 326 and instructions 320, 328 may be stored in a location physically away from the processor but still accessible by processors 314, 322. Similarly, processors 314, 322 may include a collection of processors that can perform concurrent and / or sequential operations. Computing devices 304, 306 may each include one or more internal clocks that provide timing information, which may be used for time measurement of operations and programs run by computing devices 304, 306.

[0049] The computing devices 304, 306 may be able to communicate directly and indirectly over the network 310. The devices 304, 306 may establish listening sockets that may accept initiated connections for sending and receiving information. The network 310 itself may include a variety of configurations and protocols, including the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local area network, and a private network using one or more company-proprietary communication protocols. The network 310 may support a variety of short-range and long-range connections. The short-range and long-range connections may be made over different bandwidths, such as those typically associated with a wireless network. 2.402 GHz to 2.480 GHz, usually associated with the 2.4 GHz and 5 GHz associated with communication protocols; or using multiple communication standards, such as for wireless broadband communication Additionally or alternatively, the network 310 may also support wired connections between the computing devices 304, 306, including through various types of Ethernet connections.

[0050] although Figure 3 304 and user computing device 306, but it should be understood that aspects of the present disclosure may be implemented according to a variety of different configurations and numbers of computing devices, including in a paradigm for sequential or parallel processing, or through a distributed network of multiple devices. In some implementations, aspects of the present disclosure may be performed on a single device and any combination thereof.

[0051] Figure 4 A flow chart of an example process 400 for processing a query is depicted. The example process 400 may be performed on a system of one or more processors located in one or more locations, such as at Figure 11. The unified runtime catalog 100 may create, list, and / or manage table entries 106 and / or governance policies 108 through a data lake window 102 and a data warehouse window 104. The unified runtime catalog 100 may receive data definition language (DDL) from a data lake query engine 110 and a data warehouse query engine 112. Using the DDL, the unified runtime catalog 100 may return table definitions with metadata to the data lake query engine 110 and the data warehouse query engine 112 to process queries. The table definitions allow the data lake query engine 110 and the data warehouse query engine 112 to read and write tables stored in a data lake storage 114 or a data warehouse storage 116.

[0052] As shown in block 410, the unified runtime catalog 100 can receive a first request from a first query engine to generate a cloud storage table of a cloud storage system. The cloud storage table can correspond to a table of a data lake storage device 114 or a data warehouse storage device 116. The first query engine can correspond to one of the data lake query engines 110 or one of the data warehouse query engines 112.

[0053] As shown in block 420, the unified runtime catalog 100 may generate a cloud storage table in a format compatible with the first query engine. For example, if the first query engine corresponds to one of the data lake query engines 110, the unified runtime catalog 100 may generate a cloud storage table in a format compatible with the data lake query engine 110, such as by being stored in the data lake storage 114. Similarly, if the first query engine corresponds to one of the data warehouse query engines 112, the unified runtime catalog 100 may generate a cloud storage table in a format compatible with the data warehouse query engine 112, such as by being stored in the data warehouse storage 116.

[0054] As shown in block 430, the unified runtime catalog 100 may receive a second request from a second query engine to access data from a cloud storage table to process a query. The second query engine may correspond to one of the data warehouse query engines 112 or one of the data lake query engines 110. For example, the second query engine may request a table definition to read a cloud storage table.

[0055] As shown in block 440, the unified runtime catalog 100 can determine the type of table metadata of the table entry 106 required by the second query engine to access data from the cloud storage table. The unified runtime catalog 100 can identify the format of the second query engine. For example, the unified runtime catalog 100 can determine that the second query engine corresponds to a data warehouse query engine 112 or a data lake query engine 110, and therefore the table metadata should be in a format compatible with the data warehouse query engine 112 or the data lake query engine 110, respectively.

[0056] The unified runtime catalog 100 can also determine that the second request is from a query engine that is incompatible with the format of the cloud storage table. For example, the first query engine can correspond to the data lake query engine 110 and the second query engine can correspond to the data warehouse query engine 112, or the first query engine can correspond to the data warehouse query engine 112 and the second query engine can correspond to the data lake query engine 110.

[0057] As shown in block 450, the unified runtime catalog 100 may generate table metadata for the table entry 106 as a type compatible with the second query engine. The unified runtime catalog 100 may include a translation layer that generates the table metadata using a mapping between the formats of different cloud storage systems. The mapping may include compatibility features between the format of the second query engine and the format of the cloud storage table. The compatibility features may include schemas, data types, column identifiers, and / or table attributes that control data layout as examples. For example, the unified runtime catalog 100 may use a mapping between the format of the data lake and the format of the data warehouse to generate the table metadata as a type compatible with the data warehouse query engine 112 or the data lake query engine 110, regardless of whether the cloud storage table is generated in a format compatible with the data lake query engine 110 or the data warehouse query engine 112.

[0058] As shown in block 460, the unified runtime catalog 100 may provide the table metadata of the table entry 106 to the second query engine to allow the second query engine to process the query. For example, the unified runtime catalog 100 may respond to the request for the table definition using the translated table definition native to the second query engine, thereby allowing the second query engine to read the cloud storage table.

[0059] Figure 5 A flow chart of an example process 500 for directing query processing when associated with a governance policy is depicted. The example process 500 may be performed on a system of one or more processors located in one or more locations, such as at Figure 1 . The unified runtime catalog 100 can enable a governance policy 108 for a query engine that does not support the governance policy 108 by selecting table metadata (such as table definitions) based on whether the query engine supports the governance policy 108. The table metadata can guide the query engine to access the data directly or through the unified runtime API 118. Therefore, the unified runtime catalog 100 can ensure that the governance policy 108 is uniformly enforced regardless of whether the request comes from the data lake query engine 110 or the data warehouse query engine 112.

[0060] As shown in block 510, the unified runtime catalog 100 may receive a request from a query engine to access data from a cloud storage table of a cloud storage system. The cloud storage table may correspond to a table of a data lake storage 114 or a data warehouse storage 116. The query engine may correspond to one of the data lake query engines 110 or one of the data warehouse query engines 112.

[0061] As shown in block 520, the unified runtime catalog 100 can determine that the cloud storage system is associated with one or more governance policies 108. Example governance policies 108 can include ACLs and / or FGACs, such as column security, data masking, row filtering, k-anonymity, or differential privacy. The unified runtime catalog 100 can use the table entries 106 and the governance policies 108 to determine which cloud storage systems are associated with which governance policies 108, if any.

[0062] As shown in block 530, the unified runtime catalog 100 may determine whether the query engine supports one or more governance policies 108 associated with the cloud storage system. The unified runtime catalog 100 may store a mapping of which engines support which governance policies 108 to determine whether the query engine supports the one or more governance policies 108. For example, a data warehouse query engine 112 and some data lake query engines 110 may not support FGAC, while other data lake query engines 110 may support FGAC.

[0063] If the query engine supports one or more of the governance policies 108, the unified runtime catalog 100 may instruct the query engine to retrieve data directly from the cloud storage table of the cloud storage system, as shown in block 540. If the query engine does not support one or more of the governance policies 108, the unified runtime catalog 100 may instruct the query engine to retrieve data from the cloud storage table through the unified runtime API 118, as shown in block 550.

[0064] Figure 6 A flow chart of an example process 600 for managing updates to a cloud storage table when associated with a governance policy is depicted. The example process 600 may be performed on a system of one or more processors located in one or more locations, such as at Figure 1. The unified runtime catalog 100 can manage updates from the query engine so that any updates do not affect one or more governance policies 108. The unified runtime catalog 100 can only commit changes that do not affect one or more governance policies 108, while rejecting changes that may affect one or more governance policies 108. For example, a data lake query engine 110 that is unaware of row access policies may attempt to rename a column that is mentioned in a row access policy. Renaming a column may inadvertently expose data by causing the row access policy to no longer work. The unified runtime catalog 100 can validate the updates to ensure that the updates do not affect one or more governance policies 108.

[0065] As shown in block 610, the unified runtime catalog 100 can receive a request from a query engine to update a cloud storage table of a cloud storage system. The cloud storage table can correspond to a table of a data lake storage 114 or a data warehouse storage 116. The query engine can correspond to one of the data lake query engines 110 or one of the data warehouse query engines 112.

[0066] As shown in block 620, the unified runtime catalog 100 may determine that the cloud storage systems are associated with one or more governance policies 108. The unified runtime catalog 100 may use the table entries 106 and the governance policies 108 to determine which cloud storage systems are associated with which governance policies 108, if any.

[0067] As shown in block 630, the unified runtime catalog 100 may determine whether the update to the cloud storage table affects one or more governance policies 108. The unified runtime catalog 100 may include logic to verify whether the update to the cloud storage table may adversely affect one or more of the governance policies 108 bound to the cloud storage table. For example, the logic may verify that each column that is deleted may not be referenced in one or more of the governance policies 108 bound to the cloud storage table. As another example, the logic may verify that the old name and the new name of a column rename may not be referenced in one or more of the governance policies 108 bound to the cloud storage table.

[0068] If the update does not affect one or more governance policies 108, the unified runtime catalog 100 may validate the update, as shown in block 640. If the update does affect one or more governance policies 108, the unified runtime catalog 100 may reject the update, as shown in block 650.

[0069] Aspects of the present disclosure may be implemented in digital circuits, computer-readable storage media as one or more computer programs, or a combination of one or more of the foregoing. The computer-readable storage medium may be non-transitory, for example, as one or more instructions executable by a cloud computing platform and stored on a tangible storage device.

[0070] The phrase "configured to" is used in different contexts in relation to a portion of a computer system, hardware, or computer program. When a system is said to be configured to perform one or more operations, it means that the system has appropriate software, firmware, and / or hardware installed on the system that, when in operation, causes the system to perform the one or more operations. When a piece of hardware is said to be configured to perform one or more operations, it means that the hardware includes one or more circuits that, when in operation, receive inputs and generate outputs based on the inputs and corresponding to the one or more operations. When a computer program is said to be configured to perform one or more operations, it means that the computer program includes one or more program instructions that, when executed by one or more computers, cause the one or more computers to perform the one or more operations.

[0071] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the description of the foregoing embodiments should be made in an illustrative manner rather than in a manner that limits the subject matter defined by the claims. In addition, the examples described herein and the provision of clauses expressed as "such as", "including", etc. should not be interpreted as limiting the subject matter of the claims to specific examples; rather, these examples are intended to illustrate only one of many possible embodiments. In addition, the same reference numerals in different figures may identify the same or similar elements.

Claims

1. A method for processing a query, characterized in that include: receiving, by one or more processors, a request from a query engine to access data from a cloud storage table residing in a cloud storage system; determining, by the one or more processors, a type of table metadata required by the query engine to access the data from the cloud storage table; generating, by the one or more processors, table metadata compatible with the query engine using mappings between formats of different cloud storage systems; as well as The table metadata is provided to the query engine by the one or more processors.

2. The method according to claim 1, characterized in that Also includes: Receiving, by the one or more processors, a request from a second query engine to generate the cloud storage table; as well as generating, by the one or more processors, the cloud storage table in a format compatible with the second query engine; Wherein, the query engine and the second query engine have incompatible formats.

3. The method according to claim 1 or 2, characterized in that Determining the type of table metadata also includes identifying a format for the query engine, wherein the query engine is formatted for a data lake and the cloud storage table is formatted for a data warehouse, or the query engine is formatted for a data warehouse and the cloud storage table is formatted for a data lake.

4. The method according to claim 1, characterized in that The mapping includes a mapping of compatible features between a format of the query engine and a format of the cloud storage table.

5. The method according to claim 1, characterized in that Also included is determining, by the one or more processors, that the cloud storage system is associated with one or more governance policies.

6. The method according to claim 5, characterized in that Also includes: determining, by the one or more processors, that the query engine supports the one or more governance policies; as well as The data is retrieved directly from the cloud storage table by the one or more processors.

7. The method according to claim 5, characterized in that Also includes: determining, by the one or more processors, that the query engine does not support the one or more governance policies; as well as The data is retrieved from the cloud storage table by the one or more processors through a unified runtime application programming interface (API).

8. The method according to claim 5, characterized in that Also includes: determining, by the one or more processors, that an update to the cloud storage table does not affect the one or more governance policies; as well as The update to the cloud storage table is verified by the one or more processors.

9. The method according to claim 5, characterized in that Also includes: determining, by the one or more processors, that an update to the cloud storage table affects the one or more governance policies; as well as The update to the cloud storage table is rejected by the one or more processors.

10. The method according to claim 1, characterized in that Also included is storing, by the one or more processors, the file-level metadata in a metadata management system.

11. The method according to claim 1, characterized in that Also included is storing, by the one or more processors, the lineage of the data from the cloud storage table.

12. The method according to claim 1, characterized in that Also included is supporting, by the one or more processors, unstructured data assets via an object table.

13. A system, characterized in that: include: one or more processors; as well as one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for processing a query, the operations comprising: receiving a request from a query engine to access data from a cloud storage table residing in a cloud storage system; determining a type of table metadata required by the query engine to access the data from the cloud storage table; generating table metadata compatible with the query engine using mappings between formats of different cloud storage systems; and The table metadata is provided to the query engine.

14. The system of claim 13, wherein: The operations also include: receiving a request from a second query engine to generate the cloud storage table; and generating the cloud storage table in a format compatible with the second query engine; Wherein, the query engine and the second query engine have incompatible formats.

15. The system according to claim 13 or 14, characterized in that Determining the type of table metadata also includes identifying a format for the query engine, wherein the query engine is formatted for a data lake and the cloud storage table is formatted for a data warehouse, or the query engine is formatted for a data warehouse and the cloud storage table is formatted for a data lake.

16. System according to one of claims 13 to 15, characterized in that The mapping includes a mapping of compatible features between a format of the query engine and a format of the cloud storage table.

17. System according to one of claims 13 to 16, characterized in that The operations also include determining that the cloud storage system is associated with one or more governance policies.

18. The system of claim 17, wherein: The operations also include: determining that the query engine supports the one or more governance policies and directly retrieves the data from the cloud storage table; or Determining that the query engine does not support the one or more governance policies and retrieving the data from the cloud storage table through a unified runtime application programming interface (API).

19. The system of claim 17, wherein: The operations also include: determining that the update to the cloud storage table does not affect the one or more governance policies and verifying the update to the cloud storage table; or A determination is made that an update to the cloud storage table impacts the one or more governance policies and the update to the cloud storage table is rejected.

20. A non-transitory computer-readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for processing a query, characterized in that: The operations include: receiving a request from a query engine to access data from a cloud storage table residing in a cloud storage system; determining a type of table metadata required by the query engine to access the data from the cloud storage table; generating table metadata compatible with the query engine using mappings between formats of different cloud storage systems; and The table metadata is provided to the query engine.

Images