Management tables for data lakes

Abstract

Aspects of the present disclosure relate to merging data lake openness with scalable metadata of a management table in a cloud database platform, this allows for atomicity, consistency, isolation and persistence (ACID) transactions, high performance data manipulation language (DML), higher throughput flow ingestion, data consistency, schema evolution, time travel, clustering, fine-grained security, and / or automated storage optimization. Table data is stored in cloud storage in various open source file formats, while physical metadata of the table data is stored in an extensible metadata storage system.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation of U.S. patent application No. 18 / 389,331 filed on November 14, 2023, and a continuation of U.S. patent application No. 18 / 389,337 filed on November 14, 2023. Both of these U.S. patent applications claim the benefit of the filing date of U.S. Provisional Patent Application No. 63 / 535,811 filed on August 31, 2023, and the disclosures of these two U.S. patent applications and the U.S. Provisional Patent Application are hereby incorporated herein by reference. Background Art

[0003] Data lakes are increasingly built using open file formats and stored on cloud object storage due to their relatively low cost and high durability. However, the interfaces provided by cloud storage systems are limited to single object changes and do not support multi-object transactions. This creates difficulties for systems using data lake storage to implement atomicity, consistency, isolation, and durability (ACID) transactions, snapshot consistency for reads, and strong read-after-write consistency. In addition, using open source table formats to alleviate this difficulty results in lower write throughput, lower query performance, higher infrastructure management operational overhead, limited transaction support, and a weaker security model. Summary of the invention

[0004] Aspects of the present disclosure are directed to merging data lake openness with extensible metadata for managing tables in a cloud database platform, thereby allowing atomicity, consistency, isolation, and durability (ACID) transactions, high-performance data manipulation language (DML), higher throughput streaming ingestion, data consistency, schema evolution, time travel, clustering, fine-grained security, and / or automatic storage optimization. Table data is stored in cloud storage in various open source file formats, while physical metadata for table data is stored in an extensible metadata storage system.

[0005] One aspect of the present disclosure provides a method for processing queries, the method comprising: receiving, by one or more processors, a request to write one or more data files from a query engine; writing, by one or more processors, tuples of the one or more data files to a write-optimized storage in an appendable distributed file system; converting, by one or more processors, the tuples to a columnar format optimized for reading; storing, by one or more processors, the one or more data files in a cloud storage compatible with the query engine; and submitting, by one or more processors, the write as an addition to a table transaction log stored in the appendable distributed file system. Another aspect of the present disclosure provides a system comprising: one or more processors; and one or more storage devices, the one or more storage devices being coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations according to the method for processing queries. Yet another aspect of the present disclosure provides a non-transitory computer-readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations according to the method for processing queries.

[0006] In an example, the method further comprises: compressing, by the one or more processors, the transaction log into a columnar baseline. In another example, the columnar baseline is in a format compatible with the query engine.

[0007] In yet another example, the method further comprises: performing, by one or more processors, one or more maintenance tasks based on metadata in the distributed file system. In yet another example, the one or more maintenance tasks comprise at least one of the following: garbage collection, file merging, file splitting, or file re-clustering.

[0008] In yet another example, the method further includes: converting, by the one or more processors, the table transaction log into a format compatible with the query engine; and storing, by the one or more processors, the table transaction log in a cloud storage compatible with the query engine.

[0009] In yet another example, the method further includes: exporting, by one or more processors, metadata in the appendable distributed file system to a cloud storage compatible with the query engine. In yet another example, the exported metadata is in a format compatible with the query engine. In yet another example, the export is automatically triggered in response to each addition to the table transaction log.

[0010] In yet another example, the method further includes: receiving, by one or more processors, a request from a query engine to read one or more data files in cloud storage; and reading, by one or more processors, the one or more data files directly from cloud storage. In yet another example, the method further includes: receiving, by one or more processors, a request from a query engine to read one or more data files in cloud storage; and reading, by one or more processors, the one or more data files from cloud storage via an application programming interface (API) exposing cloud storage and an attachable distributed file system. In yet another example, reading the one or more data files via the API guarantees at least one of the following: exactly once delivery, or strong read-after-write semantics. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 Depicted is a block diagram of an example management table system for a cloud storage system for a data lake and a data warehouse in accordance with aspects of the present disclosure.

[0012] Figure 2 A block diagram of an example metadata storage system is depicted in accordance with aspects of the present disclosure.

[0013] Figure 3 Depicted is a block diagram of an example computing environment implementing a management table system for a cloud storage system in accordance with aspects of the present disclosure.

[0014] Figure 4 An example process for processing queries associated with writing table data is depicted in accordance with aspects of the present disclosure.

[0015] Figure 5 An example process for processing queries associated with reading table data is depicted in accordance with aspects of the present disclosure.

[0016] Figure 6 Depicted are example processes for processing queries associated with exporting metadata in accordance with aspects of the present disclosure.

[0017] Figure 7 Depicted is a block diagram of an example streaming system that combines a scratchspace that stores write-optimized data with an object store that stores read-optimized data in accordance with aspects of the present disclosure. DETAILED DESCRIPTION

[0018] The present technology generally relates to a method for managing tables in a cloud database platform that can merge data lake openness with extensible metadata. The method can allow atomicity, consistency, isolation, and durability (ACID) transactions, high-performance data manipulation language (DML), higher throughput stream ingestion, data consistency, schema evolution, time travel, clustering, fine-grained security, and / or automatic storage optimization. The method can store table data in cloud storage in various open source file formats, while storing physical metadata of the table data (e.g., file name, statistics, row-level delete bitmap) in an extensible metadata storage system. The separation of physical metadata from table data can allow higher capacity DML and cross-table transactions.

[0019] To commit a transaction, the management table system may write file-level or row-level additions and / or deletions to an append-based transaction log in a metadata storage system. The metadata storage system may include a distributed file system (e.g., an appendable distributed file system) that supports file appending to store append-based transaction logs. The distributed file system may also support snapshots (such as the ability to atomically create a logical copy of a set of files) and / or renames (such as the ability to atomically rename a set of files). The distributed file system may be replicated to one or more data centers in one or more locations. The management table system may periodically compress the append-based transaction log into a read optimization baseline to allow for improved query efficiency. The read optimization baseline may be stored in a column-oriented format. For example, the management table system may compress the append-based transaction log every minute or every hour based on the speed at which additions and / or deletions occur. The read optimization baseline may contain column-level metadata, such as a zone map and / or a bloom filter, which the management table system may use to streamline files for future queries.

[0020] The management table system may include automated data management for continuously or periodically performing maintenance tasks on the physical metadata. As examples, the maintenance tasks may include garbage collecting files, improving storage by merging smaller files into larger files, and / or performing continuous or periodic background re-clustering.

[0021] The management table system may include a storage application programming interface (API) to support higher throughput streaming ingestion from an open source query engine. The storage API may write the ingested tuples to a write-optimized storage in a distributed file system. The write-optimized storage may be stored in an appendable file system (such as a distributed file system) in a row-oriented format. The management table system may periodically convert the ingested tuples in the write-optimized storage into read-optimized columnar data files in the cloud storage in a transactional manner. For example, the management table system may convert the ingested tuples every minute or every hour depending on the rate at which the tuples are ingested. The streamed tuples may be visible after being written because the table data may represent the union of tuples stored in the cloud storage and tuples stored in the write-optimized storage in the distributed file system, thereby allowing the atomicity guarantees of the conversion to be maintained, even for cloud storage that does not support atomic rename or snapshots.

[0022] The management table system may include an open metadata view to provide a live view of the metadata storage system in an open source table format. The open metadata view may provide compatibility with external query engines that only support open source table formats, and may avoid lock-in because clients may export table data and metadata in an open source table format. Typically, data (e.g., metadata, table data, data files) is compatible with a query engine when the query engine is arranged to open, process, and / or dispose of the format of the data, and thereby open, process, and / or dispose of the data itself (e.g., metadata, table data, data files). Accordingly, the data format is compatible with the query engine when the query engine is arranged to open, process, and / or dispose of the data format. The open metadata view may be materialized to cloud storage (such as by writing the metadata view to one or more files), or accessed through an open source compatible directory API in an open source compatible metastore. Exports may be automatic or requested by running a metadata export query. Exports may also be incremental, building on previously exported snapshots and writing only the differences, rather than materializing the entire metadata snapshot for each export.

[0023] The open source query engine can read the table data of the management table system by using open metadata views to directly access files on cloud storage. Fine-grained access control lists (ACLs) may not be enforced because the open source query engine can directly access the files. Based on eventual consistency, recently streamed tuples that are still in write-optimized storage may not be visible. Alternatively or in addition, the open source query engine can read the table data by reading the table data via the storage API. The storage API can enforce fine-grained ACLs and / or provide strong read consistency regardless of whether the streamed tuples are still in write-optimized storage.

[0024] Figure 1A block diagram of an example management table system 100 for a cloud storage system for a data lake and a data warehouse is depicted. The management table system 100 can be implemented on one or more computing devices in one or more locations. The management table system 100 can include one or more data lake query engines 102 and one or more data warehouse query engines 104. The example data lake query engine 102 can include an open source engine, and the example data warehouse query engine 104 can include a warehouse native API.

[0025] The management table system 100 may also include a cloud storage 106 for storing table data 108 and metadata snapshots 110 in various file formats (such as open source file formats). The metadata snapshot 110 may refer to an object storage that materializes an open metadata view of the logical metadata 114 and / or the physical metadata 116 to the cloud storage 106 (e.g., written to one or more files). The file format may be specific to the client utilizing the management table system 100. The management table system 100 may also include a metadata storage system 112 for storing the physical metadata 114 and / or the logical metadata 116 in a distributed file system 118. Example physical metadata 114 may include a file name, statistics, and / or a row-level delete bitmap for the table data 108. Example logical metadata 116 may include a schema, attributes, and / or permissions for the table data 108. The separation of metadata (e.g., physical metadata 114 and / or logical metadata 116) from the table data 108 may allow for higher capacity DML and cross-table transactions.

[0026] The management table system 100 may receive requests to process queries from the data lake query engine 102 and the data warehouse query engine 104. The data warehouse query engine 104 may directly access the cloud storage 106 and the metadata storage system 112 to process the query. The data warehouse query engine 104 may also periodically perform factorized shuffles to allow workers in the data warehouse query engine 104 to exchange information, such as for distributed operations like join operations. The data lake query engine 102 may access the cloud storage 106 and the metadata storage system 112 through a storage API 122 (e.g., a storage read / write API) to process the query. Alternatively or in addition, the data lake query engine 102 may directly access the cloud storage 106 and the metadata storage system 112 through an open metadata view (not shown).

[0027] Figure 2 A block diagram of an example metadata storage system 200 is depicted. The example metadata storage system 200 may be used with Figure 12. The metadata storage system 200 may include a distributed file system 202 for storing metadata, such as physical metadata and / or logical metadata. The distributed file system 202 may include an append-based transaction log 204 and a write-optimized storage 206 for processing transactions in response to query requests from a query engine. The distributed file system 202 may also include logical metadata 208. The query engine and the storage API may submit transactions to the metadata storage system 200. The metadata storage system 200 may also include a data management engine 210 for continuously or periodically performing maintenance on the metadata. The metadata storage system 200 may also include an open metadata view engine 212 for providing exportable views of the metadata storage system 200 in various formats.

[0028] The storage API may support high throughput streaming ingestion based on write requests received from the data lake query engine. The storage API may write the ingested tuples of the write request to the write-optimized storage 206 of the distributed file system 202. For write optimization, the storage device 206 may have a row-oriented format for storing tuples. Periodically, the data management engine 210 and / or the query engine may transactionally convert the ingested tuples in the write-optimized storage 206 into read-optimized data files for storage in the cloud storage. For read optimization, the data files may have a column-oriented format. As an example, based on the speed of ingesting tuples, the storage API may transactionally convert the ingested tuples every minute, every hour, or continuously. The tuples may be visible after being written because the table data may represent a union of a data file stored in the cloud storage and a data file stored in the write-optimized storage 206. Even if the cloud storage is not compatible with atomic rename or snapshots, the union may allow the atomicity guarantee of the conversion to be maintained.

[0029] To commit a transaction based on a request received from the data lake query engine, the storage API may write file-level or row-level additions and / or deletions to the append-based transaction log 204 of the distributed file system 202. Periodically, the storage API may compress the append-based transaction log into a read-optimized baseline 214 for storage in cloud storage. For read optimization, the baseline 214 may have a column-oriented format. As an example, based on the rate of write additions and / or deletions, the data management engine 210 may compress the append-based transaction log 204 every minute, every hour, or continuously. The read-optimized baseline 214 may include column-level metadata that can be used to streamline files and / or improve query efficiency. For example, the column-level metadata may include a location map and / or a Bloom filter. The write-optimized storage 206, the read-optimized baseline 214, and the append-based transaction log 204 of the distributed file system 200 allow ACID transactions with higher throughput.

[0030] Based on the request received from the data lake query engine, the storage API can be used to read the table data. The storage API can use logical metadata to enforce fine-grained access control lists (FGAC). The storage API can also provide strong consistency, whether the streamed tuples are in cloud storage or still in write-optimized storage, because the table data can represent the union of the data files of the cloud storage and the data files of the write-optimized storage.

[0031] The open metadata view engine 212 can provide a live view of the metadata storage system 200 in various table formats, such as open source formats. The open metadata view engine 212 can convert metadata to various formats and export metadata in various formats. For example, the open metadata view engine 212 can be configured to convert and export metadata in a specific format compatible with a client utilizing cloud storage. A compatible format can be a format that allows a client utilizing cloud storage (such as through a query engine) to access (e.g., read) metadata. The open metadata view engine 212 can export metadata in response to a query (such as a metadata export query received from a data lake query engine) or automatically (such as in response to each addition and / or deletion to the transaction log 204). The open metadata view engine 212 allows compatibility with a data lake query engine or a data warehouse query engine that may only support a specific table format (such as an open source table format), as well as avoiding the lock-in of metadata, where the metadata may be stuck to a specific data warehouse or storage system. The open metadata view engine 212 can materialize the open metadata view to the cloud storage (such as by writing the metadata view to one or more files), or include a format-compatible API for accessing the metadata. The format-compatible API may be an API that works with the query engine to allow clients utilizing the cloud storage to access metadata.

[0032] Based on the request received from the data lake query engine, the table data can be read using the open metadata view engine 212. The open metadata view engine 212 can directly access the files in the cloud storage. Since the data lake query engine can directly access the files through the open metadata view engine 212, FGAC may not be enforced. In addition, the open metadata view engine 212 can provide eventual consistency, so recently streamed tuples that are still being written to the optimized storage may not be visible.

[0033] The data management engine 210 may perform continuous or periodic (e.g., every minute or every hour) maintenance tasks on the distributed file system (such as on physical metadata and / or logical metadata). As an example, the maintenance tasks may include garbage collection, file merging, and / or re-clustering. For example, the data management engine 210 may delete files containing data that have been deleted from the cloud storage from the distributed file system, such as garbage collection of tuples stored in the write optimization buffer 206. As another example, the data management engine 210 may merge smaller files into larger files and / or split larger files into smaller files, such as for optimizing file size based on performance, computing power, and / or exploiting parallelism. For example, streaming smaller amounts of data tends to produce smaller files, which can then be combined and clustered to maintain performance. As yet another example, the data management engine 210 may perform background re-clustering.

[0034] Figure 3 A block diagram of an example computing environment 300 implementing a management table system 302 for a cloud storage system is depicted. The management table system 302 can be used with Figure 1 302. The management table system 302 may be implemented on one or more devices having one or more processors in one or more locations, such as in a server computing device 304. The client computing device 306 and the server computing device 304 may be communicatively coupled to one or more storage devices 308 via a network 310. The server computing device 304 and the storage device 308 may form part of a cloud computing system 312 for cloud computing services, such as infrastructure as a service (IaaS), platform as a service (PaaS), and / or software as a service (SaaS).

[0035] For example, client computing device 306 may use cloud computing system 312 as a service to provide software applications such as accounting, word processing, inventory tracking, fraud detection, file sharing, video sharing, audio sharing, communications, or gaming. As another example, client computing device 306 may access cloud computing system 312 as part of one or more operations that employ machine learning, deep learning, and / or artificial intelligence techniques to train a software application. Cloud computing system 312 may provide model parameters that may be used to update a machine learning model for a software application.

[0036] The storage device 308 may be a combination of volatile and non-volatile memory and may be located at the same or a different physical location than the computing devices 304, 306. For example, the storage device 308 may include any type of non-transitory computer-readable medium capable of storing information, such as a hard disk, a solid-state drive, a tape drive, an optical storage device, a memory card, ROM, RAM, DVD, CD-ROM, a writable memory, and a read-only memory.

[0037] The server computing device 304 may include one or more processors 314 and memory 316. The memory 316 may store information accessible by the processor 314, including instructions 318 executable by the processor 314. The memory 316 may also include data 320 that may be retrieved, manipulated, or stored by the processor 314. The memory 316 may be a type of non-transitory computer-readable medium capable of storing information accessible by the processor 314, such as volatile memory and non-volatile memory. The processor 314 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0038] Instructions 318 may include one or more instructions that, when executed by processor 314, cause one or more processors to perform actions defined by instructions 318. Instructions 318 may be stored in an object code format for direct processing by processor 314, or in other formats, including a collection of interpretable scripts or independent source code modules that can be interpreted on demand or pre-compiled. Instructions 318 may include instructions for implementing management table system 302. Management table system 302 may be executed using processor 314 and / or using other processors located remotely from server computing device 304.

[0039] Data 320 may be retrieved, stored, or modified by processor 314 according to instructions 318. Data 320 may be stored in a computer register, in a relational or non-relational database as a table with a plurality of different fields and records, or as a JSON, YAML, proto, or XML document. Data 320 may also be formatted in a computer-readable format such as, but not limited to, binary values, ASCII, or Unicode. In addition, data 320 may include information sufficient to identify the relevant information, such as a number, descriptive text, proprietary code, pointer, reference to data stored in other storage (including other network locations), or information used by a function to calculate the relevant data.

[0040] The client computing device 306 may also be configured similarly to the server computing device 304, with one or more processors 322, memory 324, instructions 326, and data 328. The client computing device 306 may also include a client input 330 and a client output 332. The client input 330 may include any suitable mechanism or technology for receiving input from a client, such as a keyboard, mouse, mechanical actuator, soft actuator, touch screen, microphone, and sensor.

[0041] The server computing device 304 may be configured to transmit data to the client computing device 306, and the client computing device 306 may be configured to display at least a portion of the received data on a display implemented as part of the client output 332. The client output 332 may also be used to display an interface between the client computing device 306 and the server computing device 304. The client output 332 may alternatively or additionally include one or more speakers, transducers or other audio outputs, tactile interfaces, or other tactile feedback that provides non-visual and non-audible information to a client of the client computing device 306.

[0042] although Figure 3 Processors 314, 322 and memories 316, 324 are shown as being located within computing devices 304, 306, but the components described herein (including processors 314, 322 and memories 316, 324) may include multiple processors and memories that can operate at different physical locations rather than within the same computing device. For example, some of data 318, 326 and instructions 320, 328 may be stored on a removable SD card, while other instructions are stored in a read-only computer chip. Some or all of data 31, 326 and instructions 320, 328 may be stored in a location physically away from the processor but still accessible by processors 314, 322. Similarly, processors 314, 322 may include a collection of processors that can perform concurrent and / or sequential operations. Computing devices 304, 306 may each include one or more internal clocks that provide timing information, which may be used for time measurement of operations and programs run by computing devices 304, 306.

[0043] The computing devices 304, 306 may be able to communicate directly and indirectly over the network 310. The devices 304, 306 may establish listening sockets that may accept initiated connections for sending and receiving information. The network 310 itself may include a variety of configurations and protocols, including the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local area network, and a private network using one or more company-proprietary communication protocols. The network 310 may support a variety of short-range and long-range connections. The short-range and long-range connections may be made over different bandwidths, such as those typically associated with a wireless network. 2.402 GHz to 2.480 GHz, usually associated with the 2.4 GHz and 5 GHz associated with communication protocols; or using multiple communication standards, such as for wireless broadband communication Additionally or alternatively, the network 310 may also support wired connections between the computing devices 304, 306, including through various types of Ethernet connections.

[0044] although Figure 3304 and user computing device 306, but it should be understood that aspects of the present disclosure may be implemented according to a variety of different configurations and numbers of computing devices, including in a paradigm for sequential or parallel processing, or through a distributed network of multiple devices. In some implementations, aspects of the present disclosure may be performed on a single device and any combination thereof.

[0045] Figure 4 An example process 400 for processing queries associated with writing table data is depicted. The example process 400 may be performed on a system of one or more processors in one or more locations, such as Figure 1 Executed on the management table system 100 depicted in FIG.

[0046] As shown in block 410, the table management system 100 may be configured to receive a request from a query engine to write one or more tuples. The query engine may be a data lake query engine, such as an open source engine.

[0047] As shown in block 420, the management table system 100 may be configured to write one or more tuples to a storage device in a distributed file system. The storage device may be a write-optimized storage having a row-oriented format for storing tuples. The distributed file system may be an appendable distributed file system.

[0048] As shown in block 430, the management table system 100 may be configured to convert the tuple into one or more data files in a format compatible with the query engine. The management table system 100 may convert the tuple into a read-optimized format having a column-oriented format. The management table system 100 may periodically or continuously convert the tuple based on the speed at which the tuple is ingested. The management table system 100 may also be configured to store one or more data files in a cloud storage compatible with the query engine. Since the table data may represent a union of data files stored in the cloud storage and the storage device, the tuple may be visible after being written even if it has not yet been converted.

[0049] As shown in block 440, the management table system 100 may commit writes as additions and / or deletions to a table transaction log in a distributed file system. For example, the commit may occur exactly once. The management table system 100 may periodically or continuously compress the transaction log into a columnar baseline based on the speed at which additions and / or deletions occur. The columnar baseline may be read optimized and in a format compatible with a query engine. The columnar baseline may include column-level metadata.

[0050] Figure 5 An example process 500 for processing a query associated with reading table data is depicted. The example process 500 may be performed on a system of one or more processors in one or more locations, such as Figure 1Executed on the management table system 100 depicted in FIG.

[0051] As shown in block 510, the management table system 100 may be configured to receive a request from a query engine to read one or more data files in a cloud storage in a format compatible with the query engine. The query engine may be a data lake query engine, such as an open source engine.

[0052] As shown in block 520, the management table system 100 may be configured to determine whether to read the one or more data files directly or through a storage API. The management table system 100 may determine how to read the one or more data files based on instructions included in the request.

[0053] In response to the determination, as shown in block 530, the management table system 100 may be configured to directly read one or more data files. The management table system 100 may use an open metadata view engine to directly access one or more data files in the cloud storage. Based on direct file access, FGAC may not be enforced. In addition, based on eventual consistency, recently streamed tuples in the write-optimized storage may not be visible.

[0054] In response to the determination, the management table system 100 may be configured to read one or more data files through a storage API, as shown in block 540. The storage API may use logical metadata to enforce FGAC and provide strong consistency because the storage API may access the union of data files between cloud storage and write-optimized storage.

[0055] Figure 6 An example process 600 for processing queries associated with exporting metadata is depicted. The example process 600 may be performed on a system of one or more processors in one or more locations, such as Figure 1 Executed on the management table system 100 depicted in FIG.

[0056] As shown in block 610, the management table system 100 may be configured to receive a request from a query engine to export metadata associated with table data in a cloud storage in a format compatible with the query engine. The query engine may be a data lake query engine, such as an open source engine.

[0057] As shown in block 620, the management table system 100 may be configured to convert the metadata into a format compatible with the query engine. The management table system 100 may convert the table transaction log into a format compatible with the query engine. The management table system 100 may convert the metadata into a specific format based on the format of the query received from the query engine.

[0058] As shown in block 630, the manage table system 100 may be configured to export metadata. The manage table system 100 may store table transaction logs in a cloud storage that is compatible with the query engine.

[0059] Figure 7 A block diagram of an example streaming subsystem 700 is depicted. The example streaming subsystem 700 may be included as Figure 1 Streaming subsystem 700 may be implemented on one or more computing devices in one or more locations. Streaming subsystem 700 may include a write API 702 (as part of a streaming front end), a scratch space 704, and a streaming back end 706. Write API 702 may be used with Figure 1 The storage API 122 described in the figure may correspond to the storage API 122 described in the figure. The erase space 704 may be included as Figure 1 For example, the erase space 704 may be associated with a portion of the distributed file system 118 depicted in FIG. Figure 2 The streaming backend 706 may correspond to the write optimization buffer 206 depicted in FIG. Figure 2 Corresponding to the data management engine 210 depicted in .

[0060] The streaming subsystem 700 can provide higher throughput streaming on the object store 708 without encountering the small file problem. The small file problem can refer to the creation of a new file immediately after streaming a few records, resulting in a table with many tiny files, which adds significant time overhead for the file list and metadata overhead for each file. When reading data, these overheads limit the streaming throughput and reduce query performance. The object store 708 can be used with Figure 1 The streaming subsystem 700 may utilize the scratch space 704 to convert and output data in batches to a cloud object storage device 708 in various formats, such as an open source format.

[0061] The streaming subsystem 700 can receive queries from various query engines 710. The query engines 710 can communicate with various Figure 1 700 corresponds to the data lake query engine 102 and / or the data warehouse query engine 104 depicted in FIG. Based on the query, the streaming subsystem 700 uses the write API 702 to ingest data and store the data in a durable write-optimized storage, such as a scratch space 704 supporting an attached distributed file system using a row-oriented format. The write-optimized storage can be replicated to multiple locations for high availability.

[0062] The streaming subsystem 700 can batch convert data from one or more intermediate files in the scratch space 704 to final files for storage in various file formats in the object store. The streaming subsystem 700 can use the streaming backend 706 to batch convert the data. The metadata of the newly created object storage file can be saved in the management table system. The query engine 710 can use the read API 712 to read the data immediately after the write confirmation, because the data to be read may include data from both the object store 708 and the scratch space 704.

[0063] The streaming subsystem 700 allows row-level information to be inserted into an object store table in real time, such as row by row, by row set, and / or in batches, via a write API 702. Information may also be inserted, updated, and / or deleted via the write API 702. The streaming subsystem 700 allows the streamed data to be queried via a query engine 710 using a read API 712 immediately after delivery of the streamed data is confirmed, even before final materialization in the object store 708.

[0064] The managed table system may represent the union of tuples stored in immutable files in cloud object storage and high throughput write-optimized storage stored on-premises. Atomicity guarantees may be provided by converting files in write-optimized storage to immutable files in cloud object storage by executing transaction commits that invalidate write-optimized files and add read-optimized files to the managed table system. Atomicity guarantees may include exactly-once delivery, where each streamed record is guaranteed to be added to the table once, and strong read-after-write semantics, where streamed data is visible to queries immediately after a write confirmation or commit.

[0065] Aspects of the present disclosure may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, and / or in computer hardware (such as the structures disclosed herein, their structural equivalents, or combinations thereof). Aspects of the present disclosure may be further implemented as one or more computer programs, such as one or more modules of computer program instructions encoded on a tangible, non-transitory computer storage medium, for execution by one or more data processing devices or for controlling the operation of one or more data processing devices. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof. The computer program instructions may be encoded on an artificially generated propagated signal (such as a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0066] The term "configured" is used herein in conjunction with system and computer program components. For a system of one or more computers configured to perform specific operations or actions, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof that enables the system to perform those operations or actions. For one or more computer programs configured to perform specific operations or actions, it is meant that the one or more programs include instructions that, when executed by one or more data processing devices, cause the devices to perform those operations or actions.

[0067] The term "data processing apparatus" or "data processing system" refers to data processing hardware and encompasses a variety of devices, apparatuses, and machines for processing data, including programmable processors, computers, or combinations thereof. Data processing apparatus may include dedicated logic circuitry, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). Data processing apparatus may include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination thereof.

[0068] The term "computer program" refers to a program, software, software application, app, module, software module, script or code. A computer program can be written in any form of programming language, including compiled, interpreted, declarative or procedural languages ​​or a combination thereof. A computer program can be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for a computing environment. A computer program may correspond to a file in a file system and may be stored in a portion of a file that holds other programs or data (such as one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (such as files that store one or more modules, subroutines or code portions). A computer program may be executed on one computer, or on multiple computers located at one site or distributed over multiple sites and interconnected by a data communications network.

[0069] The term "database" refers to any collection of data. The data may be unstructured, or structured in any manner. The data may be stored on one or more storage devices in one or more locations. For example, an index database may include multiple collections of data, each of which may be organized and accessed differently.

[0070] The term "engine" refers to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. An engine may be implemented as one or more software modules or components or may be installed on one or more computers in one or more locations. A particular engine may have its own dedicated computer or computers, or multiple engines may be installed and run on the same computer or computers.

[0071] The processes and logic flows described herein may be performed by one or more computers executing one or more computer programs to perform functions by operating on input data and generating output data. The processes and logic flows may also be performed by a dedicated logic circuit system or by a combination of a dedicated logic circuit system and one or more computers.

[0072] A computer or special purpose logic circuit system that executes one or more computer programs may include a central processing unit (including a general or special purpose microprocessor) for running or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit may receive instructions and data from one or more memory devices (such as read-only memory, random access memory, or a combination thereof), and may run or execute these instructions. The computer or special purpose logic circuit system may also include or be operatively coupled to one or more storage devices (such as magnetic disks, magneto-optical disks, or optical disks) for storing data to receive data from the storage device or transfer data to the storage device. The computer or special purpose logic circuit system may be embedded in another device, such as, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS), or a portable storage device, such as a universal serial bus (USB) flash drive.

[0073] Computer-readable media suitable for storing one or more computer programs may include any form of volatile or non-volatile memory, medium, or storage device. Examples include semiconductor memory devices (e.g., EPROM, EEPROM, or flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, CD-ROM disks, DVD-ROM disks, or combinations thereof.

[0074] Aspects of the present disclosure may be implemented in a computing system that includes a back-end component (e.g., as a data server), a middleware component (e.g., an application server), or a front-end component (e.g., a client computer with a graphical user interface, a web browser or app, or any combination thereof). The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0075] A computing system may include a client and a server. The client and server may be remote from each other and interact through a communication network. The relationship of client and server arises from computer programs running on respective computers and having a client-server relationship to each other. For example, a server may transmit data (e.g., an HTML page) to a client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from the user). Data generated at the client device (e.g., the result of a user interaction) may be received from the client device at the server.

[0076] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above may be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be made in an illustrative manner rather than in a manner that limits the subject matter defined by the claims. In addition, the examples described herein and the provision of clauses expressed as "such as," "including," etc. should not be interpreted as limiting the subject matter of the claims to specific examples; rather, these examples are intended to illustrate only one of many possible embodiments. In addition, the same reference numerals in different figures may identify the same or similar elements.

Claims

1. A method for processing a query, characterized in that The method comprises: Receiving, by one or more processors, a request from a query engine to read one or more data files; retrieving, by the one or more processors, the one or more data files from a cloud storage compatible with the query engine using metadata stored in the appendable distributed file system; and The one or more data files are read, by the one or more processors, from the cloud storage that is compatible with the query engine.

2. The method according to claim 1, characterized in that Also includes: The metadata stored in the attachable distributed file system is exported, by the one or more processors, to the cloud storage in one or more formats compatible with the query engine.

3. The method according to claim 2, characterized in that The export is automatically triggered in response to one or more additions to a table transaction log stored in the appendable distributed file system.

4. The method according to claim 3, characterized in that The one or more additions to the table transaction log are in response to a request to write one or more data files to the cloud storage.

5. The method according to claim 1, characterized in that Reading the one or more data files further includes reading the one or more data files directly from the cloud storage.

6. The method according to claim 1, characterized in that The metadata is stored in a table transaction log in the appendable distributed file system.

7. The method according to claim 6, characterized in that The table transaction log is periodically compressed into a read-optimized format that is compatible with the query engine.

8. The method according to claim 1, characterized in that The query engine includes a data lake query engine.

9. The method according to claim 1, characterized in that Also includes: One or more maintenance tasks are performed by the one or more processors based on the metadata in the distributed file system.

10. The method according to claim 9, characterized in that The one or more maintenance tasks include at least one of: garbage collection, data file merging, data file splitting, or data file re-clustering.

11. A system, characterized in that: The system comprises: one or more processors; and one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for processing a query, the operations comprising: receiving a request from a query engine to read one or more data files; retrieving the one or more data files from a cloud storage compatible with the query engine using metadata stored in the appendable distributed file system; and The one or more data files are read from the cloud storage that is compatible with the query engine.

12. The system according to claim 11, characterized in that The operations also include exporting the metadata stored in the attachable distributed file system to the cloud storage in one or more formats compatible with the query engine.

13. The system of claim 12, wherein: The export is automatically triggered in response to one or more additions to a table transaction log stored in the appendable distributed file system.

14. The system of claim 13, wherein: The one or more additions to the table transaction log are in response to a request to write one or more data files to the cloud storage.

15. The system of claim 11, wherein: Reading the one or more data files further includes reading the one or more data files directly from the cloud storage.

16. The system of claim 11, wherein: The metadata is stored in a table transaction log in the appendable distributed file system.

17. The system of claim 16, wherein: The table transaction log is periodically compressed into a read-optimized format that is compatible with the query engine.

18. The system of claim 11, wherein: The operations also include performing one or more maintenance tasks based on the metadata in the distributed file system.

19. A non-transitory computer-readable medium, characterized in that The non-transitory computer readable medium is used to store instructions that, when executed by one or more processors, cause the one or more processors to perform operations for processing a query, the operations comprising: receiving a request from a query engine to read one or more data files; retrieving the one or more data files from a cloud storage compatible with the query engine using metadata stored in the appendable distributed file system; and The one or more data files are read from the cloud storage that is compatible with the query engine.

20. The non-transitory computer readable medium of claim 19, wherein: The operations also include exporting the metadata stored in the attachable distributed file system to the cloud storage in one or more formats compatible with the query engine.

Images