Database metadata in immutable storage

Abstract

The invention relates to database metadata in an immutable storage device. A method for a database system includes storing table data of a database, the table data including information in rows and columns of one or more database tables. The method includes storing metadata on an immutable storage device, the metadata including information about table data of a database. In one embodiment, variable metadata may be periodically merged in the background to create a new version of metadata files and allow for deletion of old metadata files and old data files.

Description

[0001] This application is a divisional application of an application filed on November 14, 2018, with application number 201880082019.1 and invention name “Database metadata in an immutable storage device”. Technical Field

[0002] The present disclosure relates to systems, methods, and apparatus for databases, and more particularly, to using a non-mutable storage service to store and maintain metadata. Background Art

[0003] Databases are widely used for data storage and access in computing applications. A database may include one or more tables that include or reference data that can be read, modified, or deleted using queries. A database may store small or extremely large data sets in one or more tables. The data may be accessible to individual users in an organization, such as through a website or application program interface (API), or may even be used to serve public users. Both computing resources and storage resources and their underlying architecture can play an important role in achieving ideal database performance. Summary of the invention

[0004] Aspects of the present disclosure may be implemented in one or more embodiments below.

[0005] 1) A database system, comprising: means for storing table data of a database, the table data comprising information in rows and columns of one or more database tables; and A tool for storing metadata on an immutable storage device, the metadata comprising information about table data of the database, wherein storing the metadata comprises: storing indications of added or deleted table data files, storing one or more of a file name and a storage location of one or more table data files.

[0006] 2) The database system according to claim 1, further comprising a tool for storing information about the metadata in a variable storage device, wherein the tool for storing information about the metadata comprises a tool for storing the information about the metadata in a key-value storage device, the information about the metadata comprises a version of the table data, and the information about the metadata indicates one or more metadata files including metadata corresponding to the version.

[0007] 3) The database system according to 1), further comprising a tool for creating a new metadata file in the immutable storage device in response to a change in the table data without modifying a previous metadata file, the new metadata file comprising metadata indicating the change in the table data.

[0008] 4) The database system according to 1) further includes a tool for merging metadata from two or more old metadata files into a merged metadata file, wherein the merged metadata file includes metadata reflecting changes in table data indicated in the two or more old metadata files.

[0009] 5) The database system according to 4), further comprising means for storing one or more metadata file versions, wherein the metadata file versions indicate a metadata file corresponding to a pre-merge version and a metadata file corresponding to a post-merge version.

[0010] 6) The database system according to 1), further comprising: means for receiving a query directed to a database comprising said table data; and Means for retrieving in parallel from mutable storage a plurality of uncached metadata files corresponding to the query.

[0011] 7) The database system according to 6) further includes a tool for determining a scan set of table data files based on the metadata file corresponding to the query, wherein the tool for determining the scan set reads a first metadata file before a second metadata file is completely downloaded, and the multiple uncached metadata files include the first metadata file and the second metadata file, wherein the scan set indicates one or more table data files required to execute the query.

[0012] 8) The database system according to 1), wherein the means for storing the metadata stores the metadata in a column-by-column format, and the database system further comprises: Tools for encrypting the metadata on a column-by-column basis to allow independent decryption and reading of metadata for specific columns; and A tool for storing hash values ​​of columns of metadata and comparing the stored hash values ​​to calculated hash values ​​to determine whether the metadata has been altered.

[0013] 9) A method comprising: storing table data of a database, the table data comprising information in rows and columns of one or more database tables; and Metadata is stored on an immutable storage device, the metadata including information about table data of the database.

[0014] 10) The method according to 9), wherein storing the metadata comprises storing and managing the metadata in a file on the immutable storage device.

[0015] 11) The method according to 9), wherein the metadata includes an indication of an added table data file or a deleted table data file.

[0016] 12) The method according to 9), wherein the metadata includes file information about a file including the table data, the file information including one or more of: a file name and a storage location.

[0017] 13) The method according to 9), wherein the immutable storage device comprises a cloud storage resource.

[0018] 14) The method according to 9), further comprising storing information about the metadata in a variable storage device.

[0019] 15) The method according to 14), wherein the information about the metadata is stored in a key-value store.

[0020] 16) The method according to 14), wherein the information about the metadata includes information indicating a version and indicating one or more metadata files including metadata corresponding to the version.

[0021] 17) The method according to 9), further comprising: in response to a change in the table data, creating a new metadata file in the immutable storage device without modifying a previous metadata file, the new metadata file including metadata indicating the change in the table data.

[0022] 18) The method according to 17), wherein the metadata in the new metadata file indicates addition or deletion of a table data file including the table data.

[0023] 19) The method according to 17), further comprising deleting expired metadata files.

[0024] 20) The method according to 9), further comprising merging metadata from two or more old metadata files into a merged metadata file.

[0025] 21) The method according to 20), wherein the merged metadata file includes metadata reflecting changes in table data indicated in the two or more old metadata files.

[0026] 22) The method according to 20), wherein merging comprises deleting the two or more old metadata files.

[0027] 23) The method according to 20), further comprising deleting one or more table data files not referenced by the metadata in the merged metadata file.

[0028] 24) The method according to 20), further comprising storing one or more metadata file versions, wherein the metadata file versions indicate a metadata file corresponding to a pre-merge version and a metadata file corresponding to a post-merge version.

[0029] 25) The method according to 9), further comprising: receiving a query directed to a database including the table data; and A plurality of uncached metadata files corresponding to the query are retrieved in parallel from the immutable storage device.

[0030] 26) The method according to 25) further includes determining a scan set of table data files based on the metadata file corresponding to the query, wherein determining the scan set includes reading a first metadata file before a second metadata file is completely downloaded, and the multiple uncached metadata files include the first metadata file and the second metadata file.

[0031] 27) The method according to 26), wherein the scan set indicates one or more table data files required to execute the query.

[0032] 28) The method according to 9), wherein storing the metadata comprises storing the metadata in a column-by-column format.

[0033] 29) The method according to 28), further comprising encrypting the metadata column by column to allow metadata of a specific column to be independently decrypted and read.

[0034] 30) The method according to 28) further includes: storing a hash value of a column of metadata, and comparing the stored hash value with the calculated hash value to determine whether the metadata has been changed.

[0035] 31) A database system, comprising: a table data component configured to store table data of a database, the table data including information in rows and columns of one or more database tables; and A metadata component is configured to store metadata on an immutable storage device, the metadata including information about table data of the database, wherein storing the metadata includes: storing an indication of an added table data file or a deleted table data file, storing one or more of a file name and a storage location of one or more table data files.

[0036] 32) The database system according to 31) further includes a metadata information component, which is used to store information about the metadata in a variable storage device, wherein the metadata information component stores the information about the metadata in a key-value storage device, the information about the metadata includes a version of the table data, and the information about the metadata indicates one or more metadata files including metadata corresponding to the version.

[0037] 33) The database system according to 31) further includes a metadata component, which is configured to: in response to changes in the table data, create a new metadata file in the immutable storage device without modifying the previous metadata file, and the new metadata file includes metadata indicating the changes in the table data.

[0038] 34) The database system according to 31) further includes a merging component, which is configured to merge metadata from two or more old metadata files into a merged metadata file, wherein the merged metadata file includes metadata reflecting changes in table data indicated in the two or more old metadata files.

[0039] 35) The database system according to 34), wherein the merging component stores one or more metadata file versions, wherein the metadata file versions indicate a metadata file corresponding to a pre-merge version and a metadata file corresponding to a post-merge version.

[0040] 36) The database system according to 31) further includes a scan set component, which is configured to determine a scan set of table data files required to execute a query, wherein the scan set component retrieves multiple uncached metadata files corresponding to the query from the immutable storage device in parallel.

[0041] 37) The database system according to 36), wherein the scan set component starts reading the first metadata file before the second metadata file is completely downloaded, and the plurality of uncached metadata files include the first metadata file and the second metadata file.

[0042] 38) The database system according to 31), wherein the metadata component stores the metadata in a column-by-column format, and the database system further comprises: an encryption component for encrypting the metadata column by column to allow the metadata of a specific column to be independently decrypted and read; and A hash component is used to store a hash value of a column of metadata and compare the stored hash value with the calculated hash value to determine whether the metadata has been changed.

[0043] 39) A computer readable storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to: storing table data of a database, the table data comprising information in rows and columns of one or more database tables; and Metadata is stored on an immutable storage device, the metadata including information about table data of the database.

[0044] 40) The computer-readable storage medium of 39), wherein the instructions cause the one or more processors to store the metadata by storing and managing the metadata in a file on the immutable storage device.

[0045] 41) The computer-readable storage medium according to 39), wherein the metadata includes an indication of an added table data file or a deleted table data file.

[0046] 42) The computer-readable storage medium according to 39), the metadata comprising file information about a file including the table data, the file information comprising one or more of: a file name and a storage location.

[0047] 43) The computer-readable storage medium of 39), wherein the instructions cause the one or more processors to store the metadata by storing on a cloud storage resource including the immutable storage device.

[0048] 44) The computer-readable storage medium of 39), wherein the instructions further cause the one or more processors to store information about the metadata in a variable storage device.

[0049] 45) The computer-readable storage medium of 44), wherein the instructions cause the one or more processors to store the information about the metadata in a key-value store.

[0050] 46) The computer-readable storage medium according to 44), wherein the information about the metadata includes information indicating a version and indicating one or more metadata files including metadata corresponding to the version.

[0051] 47) The computer-readable storage medium of claim 39), wherein the instructions further cause the one or more processors to create a new metadata file in the immutable storage device in response to a change in the table data without modifying a previous metadata file, the new metadata file including metadata indicating the change in the table data.

[0052] 48) The computer-readable storage medium according to 47), wherein the metadata in the new metadata file indicates addition or deletion of a table data file including the table data.

[0053] 49) The computer-readable storage medium according to 47), wherein the instructions further cause the one or more processors to delete expired metadata files.

[0054] 50) The computer-readable storage medium of 39), wherein the instructions further cause the one or more processors to merge metadata from two or more old metadata files into a merged metadata file.

[0055] 51) The computer-readable storage medium according to 50), wherein the merged metadata file includes metadata reflecting changes in table data indicated in the two or more old metadata files.

[0056] 52) The computer-readable storage medium of 50), wherein the instructions further cause the one or more processors to store one or more metadata file versions, wherein the metadata file versions indicate a metadata file corresponding to a pre-merge version and a metadata file corresponding to a post-merge version.

[0057] 53) The computer-readable storage medium according to 50), wherein the instructions further cause the one or more processors to delete the two or more old metadata files.

[0058] 54) The computer-readable storage medium according to 50), wherein the instructions further cause the one or more processors to delete one or more table data files that are not referenced by metadata in the merged metadata file.

[0059] 55) The computer-readable storage medium according to 39), wherein the instructions further cause the one or more processors to: receiving a query directed to a database including the table data; and A plurality of uncached metadata files corresponding to the query are retrieved in parallel from the immutable storage device.

[0060] 56) The computer-readable storage medium of 55), wherein the instructions further cause the one or more processors to determine a scan set of table data files based on a metadata file corresponding to the query, wherein the instructions cause the one or more processors to determine the scan set by reading a first metadata file before a second metadata file is completely downloaded, the plurality of uncached metadata files including the first metadata file and the second metadata file.

[0061] 57) The computer-readable storage medium according to 56), wherein the scan set indicates one or more table data files required to execute the query.

[0062] 58) The computer-readable storage medium of 39), wherein the instructions cause the one or more processors to store the metadata by storing the metadata in a column-by-column format.

[0063] 59) The computer readable storage medium of 58), wherein the instructions further cause the one or more processors to encrypt the metadata on a column-by-column basis to allow independent decryption and reading of metadata for a particular column.

[0064] 60) The computer-readable storage medium of 58), wherein the instructions further cause the one or more processors to store a hash value of a column of metadata and compare the stored hash value with the calculated hash value to determine whether the metadata has been altered. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Non-limiting and non-exhaustive embodiments of the present disclosure are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

[0066] Figure 1 is a block diagram illustrating a table with data stored in a file and associated metadata according to an example embodiment of the systems and methods described herein.

[0067] Figure 2 is an illustration of an example embodiment of the systems and methods described herein after adding Figure 1 A block diagram of the tables and metadata for an .

[0068] Figure 3 is an illustration of an example embodiment of the systems and methods described herein after adding to and deleting from a table. Figure 2 A block diagram of the tables and metadata for an .

[0069] Figure 4 is a block diagram illustrating a processing platform for a database system according to an example embodiment of the systems and methods described herein.

[0070] Figure 5 is a block diagram illustrating components of a database service manager according to one embodiment.

[0071] Figure 6 is a block diagram illustrating a table with data stored in a file and an associated metadata file in an immutable storage device according to an example embodiment of the systems and methods described herein.

[0072] Figure 7 is an illustration of an example embodiment of the systems and methods described herein after a table change Figure 6 A block diagram of the tables and metadata for an .

[0073] Figure 8 is a block diagram illustrating the merging of metadata files according to an example embodiment of the systems and methods described herein.

[0074] Fig. 9 is a block diagram illustrating components of a configuration and metadata manager according to one embodiment.

[0075] Fig.10 is a schematic flow chart illustrating a method for managing metadata in a database system according to one embodiment.

[0076] Fig.11 is a schematic flow chart illustrating a method for calculating a scan set according to one embodiment.

[0077] Fig.12 is a block diagram depicting an example computing device consistent with at least one embodiment of the processes and systems disclosed herein. DETAILED DESCRIPTION

[0078] The present disclosure relates to systems, methods, and devices for storing and maintaining mutable metadata using non-mutable storage services (such as cloud storage resources). Database systems store and maintain large amounts of metadata. The metadata describes the data stored in customer database tables, but is not actually the table data stored. The metadata can become very large, especially in large database tables with many customers. Current database systems have serious limitations in handling large amounts of metadata.

[0079] Current database systems store metadata in mutable storage devices and services, including main memory, file systems, and key-value stores. These devices and services allow metadata to be updated in-place with data. If a data record changes, it can be updated with the new information. The old information is overwritten. This allows databases to easily maintain mutable metadata: by updating metadata in-place.

[0080] However, these storage devices and services have limitations. The limitations are at least two-fold. First, mutable storage devices like main memory and file systems have a hard limit on storage capacity. If the size of the metadata exceeds these limits, it is not possible to store more metadata there. Second, mutable storage services like key-value stores perform poorly when reading large amounts of metadata. Reading the data is performed using range scans, which take a long time to complete. In practice, in large-scale deployments, a range scan can take many minutes or even close to an hour to complete.

[0081] These limitations make it impossible to store large amounts of metadata on existing mutable storage devices and services. Applicants have developed systems and methods for improving metadata storage and management, including storing metadata in immutable (non-mutable) storage. According to one embodiment, a method for storing or managing a database includes storing table data of the database. The table data includes information in rows and columns of one or more database tables. The method includes storing metadata on an immutable storage device. The metadata includes information about the table data of the database, but may not include the table data.

[0082] As used herein, immutable or non-mutable storage devices include storage devices where data cannot or is not allowed to be overwritten or updated in situ. For example, changes to data located in a unit or region of a storage medium can be stored as a new file in a different unit or region of the storage medium with a time stamp. Mutable storage devices can include storage devices where data is overwritten or updated in situ or is allowed to be overwritten or updated in situ. For example, when data associated with a given unit or region of a storage medium changes, the data in that given unit or region of the storage medium can be overwritten.

[0083] In one embodiment, metadata is stored and maintained on a non-mutable storage service in the cloud. These storage services may include, for example, Amazon S3®, Microsoft Azure Blob Storage®, and Google CloudStorage®. Many of these services do not allow data to be updated in place (i.e., non-mutable or immutable). Data files can only be added or deleted, but not updated. In one embodiment, storing and maintaining metadata on these services requires that for each change in metadata, a metadata file is added to the storage service. These metadata files can be periodically merged in the background into larger "compressed" or merged metadata files. Metadata file versions can be stored to indicate metadata files corresponding to compressed or merged versions and pre-compressed or pre-merged versions of metadata files. In one embodiment, merging mutable metadata in the background to create a new version of the metadata file can allow old metadata files and old data files to be deleted.

[0084] By using immutable storage devices (such as cloud storage devices), embodiments allow for storage capacity without hard limits. Using storage services in the cloud allows for virtually unlimited amounts of metadata. Reading large amounts of metadata can be much faster because metadata files can be downloaded in parallel, including pre-fetching of files. Metadata files can also be cached in the local file system so that they are not downloaded more than once. In actual usage scenarios and testing, applicants have observed a 200-fold performance improvement when reading metadata from a storage service in the cloud compared to reading the same metadata information from a mutable storage device like a key-value store.

[0085] A detailed description of systems and methods consistent with embodiments of the present disclosure is provided below. Although several embodiments are described, it should be understood that the present disclosure is not limited to any one embodiment, but encompasses many alternatives, modifications, and equivalents. In addition, although many specific details are set forth in the following description to provide a comprehensive understanding of the embodiments disclosed herein, some embodiments may be practiced without some or all of these details. In addition, for the sake of clarity, certain technical materials known in the relevant art are not described in detail to avoid unnecessarily obscuring the present disclosure.

[0086] Figure 1-3 Illustrated are example operations of a database system when table data is stored in an immutable storage device (such as a cloud resource) and metadata is stored in a mutable storage device (such as a local key-value store). Figure 6-8 Illustrated is an example operation of a database system when both table data and metadata are stored in an immutable storage device. In one example embodiment, the data in the database table is stored in a file in the cloud. The metadata surrounding the table and the file is stored in a metadata store. The metadata store can be a key-value store. Other example systems can use other technologies (such as main memory storage devices or file system storage devices) to store metadata.

[0087] Figure 1A table 102 with data stored in a file and associated metadata is shown. Table 102 is a "user" table stored in two physical files F1 and F2 in a cloud storage device 104. Table 102 includes a "uid" column and a "name" column. Files F1, F2 include data (e.g., field values) for the rows and columns of table 102. Specifically, file F1 includes the first three rows of table data (i.e., uids 1, 2, and 3, and the names Allison, Max, and Benoit), while file F2 includes the last three rows of table data (uids 4, 5, and 6, and the names Neda, Thierry, and Florian). In one embodiment, each file F1 and F2 stores data in a column-by-column format, where the values ​​of the "uid" column are in consecutive blocks, and the values ​​of the "name" column are in consecutive blocks of the corresponding file.

[0088] The file metadata is stored in the metadata storage 106. The file metadata contains the table version and information about each table data file (F1 and F2 in this example). The metadata storage 106 may include a mutable storage device (a storage device that can be overwritten or written in place), such as a local file system, system, memory, etc.

[0089] In one embodiment, file metadata is composed of two data sets: table version and file information. The table version data set includes a mapping of the table version to a list of added files and removed files. The file information contains information about each file, for example, including a summary of all rows and columns stored in the file, including the file path, file size, file key id, and a summary of all rows and columns stored in the file. In the state shown, table version V1 indicates that files F1 and F2 have been added (V1 -> Add: F1, F2). The file information displays information about F1 (F1 -> "cloud: / / path / to / file1", fileSize: 16MB, fileKeyId: 3452, a summary of rows and columns, etc.) and information about F2 (F2 -> " / / path / to / file2", fileSize: 11MB, fileKeyId: 7965, a summary of rows and columns, etc.).

[0090] Each modification of a table creates new files and new file metadata. Inserts into a table create new files. Deletions from a table remove files and, if not all rows in a file are deleted, may add new files with the remaining rows in the table. Updates remove files and replace them with new files whose rows contain the updated records.

[0091] Figure 2 The diagram shows the following after inserting a record into the "user" table 102. Figure 1For example, when a record (7, "Difei") is inserted into the "user" table, the data warehouse creates a new file F3 in cloud storage 104 containing the record. In addition, the file metadata in metadata storage 106 has been updated to include a new table version V2 and information about F3. Table version V2 records that file F3 has been added. The file information includes the file path, account, creation timestamp, file size, and a summary of all rows and columns storing file F3.

[0092] Figure 3 The diagram shows the following after deleting a record in the "User" table 102. Figure 2 Tables and metadata for files in the cloud. For example, when a record (4, "Neda") is deleted from the "Users" table, the warehouse can create a new file F4 containing only two records (5, "Thierry") and (6, "Florian"). File F2 can be deleted from the cloud. File F4 may be the same as the previous file F2, except that the row with uid "4" has been deleted. The new file F4 is stored in the cloud, and the file metadata is updated with a new table version V3 and file information about file F4. V3 indicates that file F4 has been added and file F2 has been deleted.

[0093] When retrieving data from a table, the data warehouse can compute a scan set of all files that need to be read. The scan set is an aggregation of all added files except for files that have been removed. The scan set can be computed using table versions. When selecting data from a table at the current time, the scan set is computed using all table versions up to the latest table version. When selecting data from a table at an earlier time, the scan set is computed using all table versions up to the current table version at the specified time. This technique of computing a scan set for any given time may be referred to herein as "time travel." For example, when a user (e.g., Figure 4 404) after V3 was implemented Figure 3 When you select data from the Users table in the Database Service Manager (for example, Figure 4 The database service manager 404 in 404 uses the table versions V1, V2, and V3 to calculate the scan set. The scan set is an aggregation of all added files F1, F2, F3, and F4 except the deleted file F2. Therefore, at the current time, the scan set contains files F1, F3, and F4.

[0094] As another example, when data is selected at an earlier time when table version V2 is the current version, table versions V1 and V2 are used to compute the scan set. The scan set is an aggregation of all added files F1, F2, F3. Since there are no removed files, the scan set contains files F1, F2, F3. In one embodiment, the scan set can be streamlined using file information. For example, a summary of the rows and columns of the files can be used to streamline the files from the scan set because the contents of these files will not be needed to compute the query results.

[0095] The above example method of storing file metadata in metadata storage 106 has limitations. It takes up too much space and results in slow performance. In practice, file metadata for hundreds of millions of files can result in terabytes of file metadata. This results in slow performance when computing scan sets and reducing scan sets. The embodiments disclosed herein overcome one or more of these limitations. Storing and maintaining such (mutable) metadata on (immutable) cloud storage allows the database system to have virtually unlimited storage capacity and faster metadata retrieval.

[0096] In one embodiment, metadata may be stored in a metadata file in an immutable storage device. In one embodiment, the system may write a metadata file to a cloud storage device for each modification of a database table. In one embodiment, the system may download and read metadata files to calculate scan sets. Metadata files may be downloaded in parallel to improve scan set calculations. In one embodiment, the system may periodically merge metadata files in the background. In one embodiment, performance improvements may be included, including prefetching, caching, columnar layout, etc. In addition, security improvements including encryption and integrity checking may also be achieved by metadata files having a columnar layout.

[0097] Go to Figure 4 , a block diagram illustrating a processing platform 400 for providing database services according to one embodiment is shown. The processing platform 400 includes a database service manager 402 that can be accessed by multiple users 404, 406, and 408. The database service manager 402 may also be referred to as a resource manager or a global service in this article. In some implementations, the database service manager 402 can support any number of users who wish to access the data or services of the processing platform 400. For example, the users 404-408 may include end users who provide data storage and retrieval queries and requests, system administrators who manage the systems and methods described herein, software applications that interact with databases, and other components / devices that interact with the database service manager 402.

[0098] The database service manager 402 may provide various services and functions to support the operation of the systems and components within the processing platform 400. The database service manager 402 may access metadata stored in association with data stored throughout the data processing platform 400. The database service manager 402 may use metadata to optimize user queries. In some embodiments, the metadata includes a summary of data stored in a remote data storage system and data available from a local cache (e.g., a cache within one or more clusters of the execution platform 412). In addition, the metadata may include information about how the data is organized in the remote data storage system and the local cache. The metadata allows systems and services to determine whether a piece of data needs to be processed without loading or accessing the actual data from a storage device.

[0099] As part of the data processing platform 400, metadata can be collected when changes are made to data using a data manipulation language (DML), which can be performed by any DML statement. Examples of manipulating data can include, but are not limited to, selecting, updating, changing, merging data, and inserting data into a table. As part of the processing platform 400, files can be created, and metadata can be collected on a per-file and per-column basis. This collection of metadata can be performed during data ingestion, or the collection of metadata can be performed as a separate process after the data is ingested or loaded. In an implementation, metadata can include the number of distinct values; the number of null values; and the minimum and maximum values ​​of each file. In an implementation, metadata can further include string length information and character ranges in the string.

[0100] In one embodiment, at least a portion of the metadata is stored in an immutable storage device. For example, the metadata may be stored on the storage platform 414 together with the table data. In one embodiment, the same or separate cloud storage resources as those used for the table data may be allocated and used for the metadata. In one embodiment, the metadata may be stored in a local immutable storage device. In one embodiment, information about metadata in an immutable storage device or information about metadata files stored in an immutable storage device is stored in a mutable storage device 410. Information about metadata may be referenced to locate and access metadata stored in an immutable storage device. In one embodiment, a system having a metadata storage device may be reconfigured so that the metadata storage device is instead used to store information about metadata files located in an immutable storage device.

[0101] The database service manager 402 also communicates with an execution platform 412, which provides computing resources for performing various data storage and data retrieval operations. The execution platform 412 may include one or more computing clusters. The execution platform 412 communicates with one or more data storage devices 416, 418, and 420 as part of the storage platform 414. Figure 4 4, and 420. In some embodiments, the data storage devices 416, 418, and 420 are cloud-based storage devices located in one or more geographic locations. For example, the data storage devices 416, 418, and 420 may be part of a distributed storage device system of a public cloud infrastructure or a private cloud infrastructure, or any other manner. The data storage devices 416, 418, and 420 may include a hard disk drive (HDD), a solid state drive (SSD), a storage cluster, or any other data storage technology. In addition, the storage platform 414 may include a distributed file system (such as a Hadoop distributed file system (HDFS)), an object storage system, and the like.

[0102] In some embodiments, the communication links between the database service manager 402 and the users 404-408, the variable storage device 410 for information about metadata files (i.e., metadata file metadata), and the execution platform 412 are implemented via one or more data communication networks, and various tasks can be assigned so that user requests can be optimized. Similarly, the communication links between the execution platform 412 and the data storage devices 416-420 in the storage platform 414 are implemented via one or more data communication networks. These data communication networks can utilize any communication protocol and any type of communication medium. In some embodiments, the data communication network is a combination of two or more data communication networks (or subnets) coupled to each other. In an alternative embodiment, these communication links are implemented using any type of communication medium and any communication protocol.

[0103] Database service manager 402, variable storage device 410, execution platform 412, and storage platform 414 are Figure 4404-408 and the data processing platform 400. In the embodiment described, the data processing platform 400 is shown as a separate component. However, each of the database service manager 402, the variable storage device 410, the execution platform 412, and the storage platform 414 can be implemented as a distributed system (e.g., distributed on multiple systems / platforms at multiple geographical locations), or can be combined into one or more systems. In addition, each of the database service manager 402, the variable storage device 410, the execution platform 412, and the storage platform 414 can be scaled up or down (independently of each other) according to changes in requests received from users 404-408 and the changing needs of the data processing platform 400. Therefore, in the described embodiment, the data processing platform 400 is dynamic and supports regular changes to meet current data processing needs.

[0104] Figure 5 A block diagram depicting components of a database service manager 402 according to one embodiment is illustrated. The database service manager 402 includes an access manager 502 and a key manager 504 coupled to a data storage device 506. The access manager 502 handles authentication and authorization tasks for the system described herein. The key manager 504 manages the storage and authentication of keys used during authentication and authorization tasks. The request processing service 508 manages received data storage requests and data retrieval requests. The management console service 510 supports access to various systems and processes by administrators and other system management personnel.

[0105] The database service manager 402 also includes a SQL compiler 512, a SQL optimizer 514, and a SQL executor 516. The SQL compiler 512 analyzes SQL queries and generates execution code for the queries. The SQL optimizer 514 determines the best method to execute the query based on the data that needs to be processed. The SQL executor 516 executes the query code for the query received by the database service manager 402. The query scheduler and coordinator 518 sends the received query to the appropriate service or system for compilation, optimization, and scheduling to the execution platform 512. The virtual warehouse manager 520 manages the operation of multiple virtual warehouses.

[0106] In addition, the database service manager 402 includes a configuration and metadata manager 522 that manages information related to data stored in remote data storage devices and local caches. Monitor and workload analyzer 524 monitors the processes executed by the database service manager 402 and manages the distribution of tasks (e.g., workloads) on execution nodes and virtual warehouses in the execution platform 412. The configuration and metadata manager 522 and the monitor and workload analyzer 524 are coupled to the data storage device 526. In one embodiment, the configuration and metadata manager 522 collects, stores, and manages metadata in immutable storage resources. In one embodiment, updates to the metadata result in new files and are not updated in place.

[0107] As discussed herein, a metadata file may include a file containing metadata for modifications (e.g., each modification) to any database table in a data warehouse. A modification to a database table may generate one or more metadata files, typically just a single metadata file. In one embodiment, the metadata file contains the following information: information about the metadata file, including a version number; a list of all added table data files; a list of deleted table data files; and information about each added table data file, including a file path, file size, file key id, and a summary of all rows and columns stored in the table data file.

[0108] In one embodiment, the content of the metadata file may change over time. If the format or content of the metadata file changes, the version number of the metadata file may be incremented. In one embodiment, the metadata repository (or other mutable data storage resource) stores only information about the metadata files (stored in the immutable storage device), and does not store information about the table data files. In practice, the information about the metadata files stored in the metadata repository (or other mutable storage device) is very limited and may contain data for thousands of metadata files. In one embodiment, information about up to 30,000 metadata files may be stored in the metadata file. This greatly reduces the amount of storage required in the metadata repository or other mutable storage device.

[0109] In one embodiment, the system writes a metadata file to the cloud storage device for each modification to a database table (e.g., a modification to a table data file). In addition to adding and deleting files, each modification to a database table in the data warehouse also generates one or more metadata files. Typically, a modification creates a single metadata file. However, if the modification to the table is large (e.g., an insert to the table that generates a very large number of files), it may result in the creation of multiple metadata files. Figure 6-12 Further operations of the configuration and metadata manager 522 are discussed further.

[0110] The database service manager 402 also includes a transaction management and access control module 528, which manages various tasks and other activities associated with the processing of data storage requests and data access requests. For example, the transaction management and access control module 528 provides consistent and synchronized access to data by multiple users or systems. Since multiple users / systems can access the same data at the same time, changes made to the data can be synchronized to ensure that each user / system works with the current version of the data. The transaction management and access control module 528 provides control of various data processing activities at a single centralized location in the database service manager 402.

[0111] Figure 6 A table 602 is shown having Figure 1 The user's table 602 is shown as having table data stored in table data files F1 and F2 in the cloud storage device 604, similar to Figure 1 However, metadata about the table data files is also stored in the metadata file MF1 in the cloud storage device 604. The metadata file MF1 contains a list of the added files F1 and F2, including all file information about these files. Figure 1 The file information in the key-value store in the embodiment of is in the metadata file (eg, MF1). Figure 6 At the time point shown, there is no file indicated to be deleted in the metadata file MF1. The metadata storage device 606 only stores the table version V1 mapped to the metadata file MF1 and the information about the metadata file MF1. The information about the metadata file MF1 includes the file path of MF1 and may include more information. Therefore, both the table data file and the metadata file are stored in the cloud storage device, while the information about the metadata file is stored in the metadata storage device 606 or other local and / or variable storage devices.

[0112] Figure 7 Figure 1 shows the data after adding a record (7, "Difei") and deleting a record (4, "Neda"). Figure 6 The first modification (inserting uid "7" and name "Difei") stores file F3 and metadata file MF2 in the cloud. MF2 lists the added file F3, including all file information about F3. Metadata storage device 606 is updated with table version V2 mapped to MF2 and information about MF2. The second modification (deleting uid "4" and name "Neda") stores file F4 and metadata file MF3 in cloud storage device 604. MF3 lists the added table data file F4, including all file information of F4, and also lists the deleted table data file of F2.

[0113] The storage of metadata files MF1, MF2, and MF3 in the cloud storage device 604 or the immutable storage device allows for increased metadata storage capacity. For example, all metadata about table data files F1, F2, F3, and F4 are found in metadata files MF1, MF2, and MF3 within the cloud storage device 604. Metadata (information about metadata) about metadata file MF1, which is much smaller in size, is stored in a key-value store, a mutable storage device, and / or a local storage device.

[0114] In one embodiment, the data warehouse computes the scan set of files that must be read to answer the query. The scan set is computed using table versions. Given a set of table versions, the data warehouse reads information about the corresponding metadata files from the metadata store. It then downloads the metadata files from the cloud and reads lists of added and deleted files. Using these lists, it computes the scan set. Using file information stored in the metadata files (e.g., information about rows and columns), the scan set can be pruned.

[0115] For example, when Figure 7 When selecting data from the "User" table at the time shown, the table versions V1, V2, and V3 are used to calculate the scan set. The warehouse reads information about the corresponding metadata files MF1, MF2, and MF3. It downloads these metadata files from the cloud. Files can be downloaded in parallel. In one embodiment, the database service manager 402 can start reading one of the files even if the other files have not been completely downloaded. From the aggregate list of added files F1, F2, F3, and F4, it removes the deleted file F2. Therefore, the resulting scan set will be F1, F3, and F4. These files (or their sub-parts) can be retrieved by the execution node that executes the query.

[0116] In one embodiment, metadata files are periodically merged in the background. The merging or "compression" of metadata files aggregates all added files from all metadata files and removes all deleted files from the list. The merge creates one or more compressed metadata files that contain only the resulting list of added files, including all file information for those files. The purpose of the merge is twofold. First, many metadata files are compressed into a much smaller set of metadata files for faster downloading and reading. Second, once old metadata files are removed, files that are no longer referenced in the compressed metadata files can be removed from the cloud.

[0117] Metadata file versions distinguish different sets of metadata files. A compressed file in a metadata file version is a merger of all metadata files of the previous metadata file version. New metadata files are always registered under the latest metadata file version. After the old metadata files are merged, the old metadata files can be deleted from the cloud storage. All files that are not referenced in the compressed file may be deleted once they are no longer referenced in any metadata file.

[0118] Figure 8 It is a graphic Figure 7 . Specifically, metadata files MF1, MF2, and MF3 are shown as being merged into a compressed metadata file MF4. Metadata file MF4 contains only the added files F1, F3, F4 because F2 is deleted in MF3. MF4 also contains all file information of F1, F3, and F4. In one embodiment, metadata file version MF V3 is created and MF4 is registered under MF V3. A new metadata file MF5 is registered under the latest metadata file version MF V3. MF5 corresponds to table version V4 ( Figure 7 ). Table version V3 can point to MF1, MF2, and MF3 of MFV1 or to MF4 of MF V3, as they will produce exactly the same scan set. As shown, the creation of the merged metadata file MF4 allows one file to perform operations that previously occupied three files. In one embodiment, an indication of the metadata file version can be stored after the merge is completed so that the version before the merge can still be determined or accessed. All subsequent changes to the table data may be reflected based on MF4 or a subsequent version. Therefore, if desired or if MF1, MF2, and MF3 represent versions that no longer need to be maintained (for example, for the purpose of a "time travel" feature), MF1, MF2, and MF3 can be deleted.

[0119] The scan set that builds a table version uses only metadata files of a single metadata file version. The metadata file version to use is the largest metadata file version that is less than or equal to the given table version. For example, building Figure 7 The scan set for table version V3 in uses metadata file version MF V3 because it is the largest metadata file version less than or equal to V3. Figure 7 , Table 1 provides a list of metadata files that must be read when building a scan set for a given table version:

[0120] Table 1

[0121] In one embodiment, the merging of metadata files occurs in a background process in the data warehouse without any impact on user workload. New metadata files can be added while the compressed files are being calculated. Only when the compressed files have been uploaded to the cloud can they be used to calculate the scan set.

[0122] Immutable storage of metadata can achieve various performance improvements. In one embodiment, metadata files are prefetched. For example, when downloading a collection of metadata files, the data warehouse downloads the metadata files in parallel in the background before the process opens the metadata files. Prefetching improves the read time of the metadata files because when the process wants to open the metadata files, the metadata files may have been downloaded using prefetching.

[0123] In one embodiment, the metadata file is cached. The metadata file may be cached on the local file system of the process. Even if the metadata file is read by many different processes sharing the same file system, the metadata file may be downloaded only once. If the cache space is insufficient, the old cached metadata file may be deleted from the cache. In this case, the metadata file may be downloaded again as needed.

[0124] In one embodiment, the metadata file has a columnar layout. The file information in the metadata file is stored in a columnar layout. This means that the format of the metadata file is not row-by-row, but column-by-column. If a process reads information about one column in the metadata file, it only needs to read a single contiguous block of bytes. In one embodiment, each block of bytes is compressed using a standard compression algorithm ("gzip"). Both techniques improve read performance.

[0125] Security improvements are also achieved in some embodiments. In one embodiment, the metadata file is encrypted using a separate file key. Within the metadata file, columns can be individually encrypted using AES-CTR mode with different starting counters. This allows the database system to read individual columns from the metadata file because the individual columns can be decrypted without having to decrypt the entire file at once. Encryption improves security because no one can read the metadata file without the correct file key.

[0126] To verify that the metadata file has not been altered, the system can store a column hash value for each column in the metadata file. Before decrypting the data, the system compares the hash value of the encrypted column with the stored hash value for that column of the metadata file. If the hash values ​​do not match, the metadata file must have been altered. This improves security because changes to the metadata file are detected by the database system.

[0127] Fig. 9902-914 is a schematic block diagram illustrating the components of a configuration and metadata manager 522 according to an embodiment. The configuration and metadata manager 522 can collect, store, and manage metadata about table data files and metadata about metadata files. The configuration and metadata manager 522 includes a table data component 902, a metadata component 904, a metadata information component 906, a merge component 908, a scan set component 910, an encryption component 912, and a hash component 914. Components 902-914 are given only as examples and may not be included in all embodiments. In fact, some embodiments may include only one component in components 902-914, or any combination of two or more components. For example, some of the components may be located outside or separated from the configuration and metadata manager 522, such as within a database service manager 402 or a processing platform 400. In addition, components 902-914 may include hardware, computer-readable instructions, or a combination of the two to perform functions and provide structures discussed herein.

[0128] Table data component 902 stores table data of a database, which includes information in rows and columns of one or more database tables. Table data component 902 can store table data in table data files within a storage resource. Example storage resources include cloud storage devices and / or immutable storage devices. In one embodiment, storage resources for storing table data files can be dynamically allocated to accommodate increases or decreases in storage requirements. Table data component 902 can manage and store table data by causing data to be stored or updated in a remote resource (e.g., a cloud storage resource or service).

[0129] The metadata component 904 stores the metadata on an immutable storage device. The metadata may include information about or describing the table data of the database stored by the table data component 902. In one embodiment, the metadata file may include metadata such as an indication of an added table data file or a deleted table data file. The metadata may include file information for the table data file, the file information including one or more of a file name and a storage location. In one embodiment, the metadata may be stored in a file on the same cloud storage resource as the table data. In one embodiment, the metadata component 904 may cause the metadata to be stored in a metadata file in a column-by-column format in a remote cloud storage device.

[0130] The metadata component 904 may also collect and manage the storage of metadata in metadata files on the immutable storage device. The metadata component 904 may create a new metadata file in the immutable storage device in response to changes in the table data without modifying the previous metadata file. The new metadata file may include metadata indicating changes in the table data. In one embodiment, the metadata in the new metadata file indicates the addition or deletion of a table data file including the table data. The metadata component 904 may also delete expired metadata files. Expired metadata files may include files that are earlier than a specific stage and are not referenced in the metadata information stored by the metadata information component 906.

[0131] The metadata information component 906 stores and manages information about metadata in a mutable storage device. Information about metadata (metadata about metadata files) can be stored in a local mutable storage device and / or a metadata storage device (or a storage device previously referred to as a metadata storage device). However, in one embodiment, the information about metadata only includes information about metadata files, and does not include metadata about table data files. Therefore, all table data metadata can be located in an immutable storage device. In one embodiment, information about metadata can be stored and updated in situ. For example, in one embodiment, information about metadata is stored in a key-value storage device. Information about metadata includes information indicating a version and information indicating one or more metadata files including metadata corresponding to the version.

[0132] The merging component 908 merges or compresses metadata from two or more old metadata files into a merged metadata file. In one embodiment, the merged metadata file includes metadata reflecting table data changes indicated in the two or more old metadata files. In one embodiment, the merging component 908 deletes two or more old metadata files. The merging component 908 may delete one or more table data files referenced by metadata in the metadata files that are not merged.

[0133] The scan set component 910 can calculate the scan set about the query. In one embodiment, the database system can receive a query pointing to a database including table data. The scan set component can retrieve multiple uncached metadata files, or cause another component to do so. The metadata file can include a metadata file corresponding to the query. In one embodiment, the scan set component downloads the metadata files in parallel from an immutable storage device. In one embodiment, the scan set component determines the scan set by reading the first metadata file before the second metadata file is completely downloaded. This can improve the speed of calculating the scan set because the processing and downloading of metadata can be performed file by file or in blocks. Therefore, before the database system starts calculating the scan set, the database system does not need to wait for all files to download, and it can calculate the scan set when the metadata file (from the cache or immutable storage device) is retrieved. In one embodiment, the scan set indicates one or more table data files required to execute the query.

[0134] The encryption component 912 is configured to encrypt table data and metadata. In one embodiment, the encryption component 912 encrypts metadata column by column to allow independent decryption and reading of metadata for a specific column.

[0135] The hash component 914 calculates and stores the hash value of the column. For example, when creating a metadata file, the hash component 914 can calculate the hash value of each column in the metadata file and store the hash value. Later, when the column in the file is accessed, the hash component 914 can calculate the hash value and compare it with the stored hash value. If the hash values ​​are different, the hash component 914 can determine that the metadata in the column has been changed.

[0136] Fig.10 is a schematic flow diagram illustrating an example method 1000 for managing metadata in a database system. The method 1000 may be performed by the configuration and metadata manager 522, the database service manager 402, the processing platform 400, and / or other services or platforms.

[0137] The method 1000 begins, and a table data component 902 stores 1002 table data of a database, the table data including information in rows and columns of one or more database tables. A metadata component 904 stores 1004 metadata on an immutable storage device, the metadata including information about the table data of the database. The metadata component 904 creates 1006 a new metadata file in the immutable storage device in response to a change in the table data, without modifying the previous metadata file. The new metadata file includes metadata indicating the change in the table data. A merge component 908 merges 1008 metadata from two or more old metadata files into a merged metadata file.

[0138] Fig.11is a schematic flow diagram illustrating an example method 1000 for computing scan sets in a database system. The method 1100 may be performed by the configuration and metadata manager 522, the database service manager 402, the processing platform 400, and / or other services or platforms.

[0139] The method 1100 begins, and the database system receives 1102 a query directed to a database including table data. The scan set component 910 identifies 1104 one or more relevant metadata files. For example, the scan set component 910 may identify 1104 the relevant metadata files based on information about metadata files stored in a metadata storage device or an immutable storage device. The scan set component 910 retrieves 1106 multiple uncached metadata files corresponding to the query from the immutable storage device in parallel. For example, if there are multiple metadata files required to calculate the scan set (but not in the cache), the multiple metadata files may be downloaded in parallel. The scan set component 910 reads 1108 a first metadata file before a second metadata file is completely downloaded to determine the scan set. For example, the scan set component 910 does not need to wait until all metadata files are downloaded before starting to calculate the scan set, it can start calculating the scan set when the files are retrieved / downloaded. The scan set can be provided to an execution node and / or used to retrieve table data files or retrieve information required for processing the query.

[0140] Fig.12 1 is a block diagram depicting an example computing device 1200. In some embodiments, computing device 1200 is used to implement one or more systems and components discussed herein. For example, computing device 1200 may include configuration and metadata manager 522, database service manager 402, processing platform 400, and / or any other component or system discussed herein, or may be a part of them. As another example, components, systems, or platforms discussed herein may include one or more computing devices 1200. In addition, computing device 1200 may interact with any system and component described herein. Accordingly, computing device 1200 may be used to perform various processes and tasks, such as those processes and tasks discussed herein. Computing device 1200 may be used as a server, client, or any other computing entity. Computing device 1200 may be any computing device in various computing devices, such as, desktop computers, notebook computers, server computers, handheld computers, tablet computers, etc.

[0141] The computing device 1200 includes one or more processors 1202, one or more memory devices 1204, one or more interfaces 1206, one or more mass storage devices 1208, and one or more input / output (I / O) devices 1210, all of which are coupled to a bus 1212. The processor 1202 includes one or more processors or controllers that execute instructions stored in the memory device 1204 and / or the mass storage device 1208. The processor 1202 may also include various types of computer-readable media, such as cache memory.

[0142] The memory device 1204 includes various computer-readable media such as volatile memory (eg, random access memory (RAM)) and / or nonvolatile memory (eg, read-only memory (ROM)). The memory device 1204 may also include a rewritable ROM such as a flash memory.

[0143] The mass storage device 1208 includes various computer-readable media, such as tapes, disks, optical disks, solid-state memory (e.g., flash memory), etc. Various drives may also be included in the mass storage device 1208 to enable reading from and / or writing to various computer-readable media. The mass storage device 1208 includes removable media and / or non-removable media.

[0144] I / O devices 1210 include various devices that allow data and / or other information to be entered into or retrieved from computing device 1200. Example I / O devices 1210 include a cursor control device, a keyboard, a keypad, a microphone, a monitor or other display device, a speaker, a printer, a network interface card, a modem, a lens, a CCD or other image capture device, and the like.

[0145] Interfaces 1206 include various interfaces that allow computing device 1200 to interact with other systems, devices, or computing environments. Example interfaces 1206 include any number of different network interfaces, such as interfaces to a local area network (LAN), a wide area network (WAN), a wireless network, and the Internet.

[0146] The bus 1212 allows the processor 1202, memory device 1204, interface 1206, mass storage device 1208, and I / O device 1210 to communicate with each other and with other devices or components coupled to the bus 1212. The bus 1212 represents one or more of several types of bus structures, such as a system bus, a PCI bus, an IEEE 1394 bus, a USB bus, etc.

[0147] Example The following examples relate to further embodiments.

[0148] Example 1 is a method that includes storing table data of a database, the table data including information in rows and columns of one or more database tables. The method includes storing metadata on an immutable storage device, the metadata including information about the table data of the database.

[0149] In Example 2, storing metadata as in Example 1 includes storing and managing the metadata in a file on the immutable storage device.

[0150] In Example 3, the metadata as in any of Examples 1-2 includes an indication of an added table data file or a deleted table data file.

[0151] In Example 4, the metadata as in any one of Examples 1-3 includes: file information for a file including the table data, the file information including one or more of a file name and a storage location.

[0152] In Example 5, the immutable storage device as in any one of Examples 1-4 includes a cloud storage resource.

[0153] In Example 6, the method as in any one of Examples 1-5 includes storing information about the metadata in a mutable storage device.

[0154] In Example 7, information about the metadata as in Example 6 is stored in a key-value store.

[0155] In Example 8, the information about metadata as in any one of Examples 6-7 includes information indicating a version and information indicating one or more metadata files including metadata corresponding to the version.

[0156] In Example 9, the method as described in any one of Examples 1-8 further includes: in response to changes in the table data, creating a new metadata file in the immutable storage device without modifying the previous metadata file, the new metadata file including metadata indicating the changes in the table data.

[0157] In Example 10, the metadata in the new metadata file in Example 9 indicates addition or deletion of a table data file including table data.

[0158] In Example 11, the method as described in any one of Examples 9-10 further includes deleting expired metadata files.

[0159] In Example 12, the method as described in any one of Examples 1-11 further includes: merging metadata from two or more old metadata files into a merged metadata file.

[0160] In Example 13, the merged metadata file of Example 12 includes metadata reflecting changes in table data indicated in two or more old metadata files.

[0161] In Example 14, the merging in any one of Examples 12-13 includes deleting two or more old metadata files.

[0162] In Example 15, the method as described in any one of Examples 12-14 further includes: deleting one or more table data files referenced by metadata in the metadata file that is not merged.

[0163] In Example 16, the method as in any one of Examples 1-15 further includes receiving a query directed to a database including table data, and retrieving a plurality of uncached metadata files corresponding to the query from an immutable storage device in parallel.

[0164] In Example 17, the method described in Example 16 also includes determining a scan set of table data files based on the metadata file corresponding to the query, wherein determining the scan set includes reading a first metadata file before a second metadata file is completely downloaded, and the plurality of uncached metadata files include the first metadata file and the second metadata file.

[0165] In Example 18, the scan set of Example 17 indicates one or more table data files required to execute the query.

[0166] In Example 19, storing the metadata as in any of Examples 1-18 includes storing the metadata in a column-by-column format.

[0167] In Example 20, the method of Example 19 further comprises encrypting the metadata column by column to allow independent decryption and reading of metadata for specific columns.

[0168] In Example 21, the method of any one of Examples 19-20 further includes storing a hash value of the column of metadata, and comparing the stored hash value with the calculated hash value (the most recently calculated hash value) to determine whether the metadata has been changed.

[0169] Example 22 is an apparatus comprising means for performing the method of any of Examples 1-21.

[0170] Example 23 is a machine-readable storage device comprising machine-readable instructions that, when executed, implement the method of any of Examples 1-22 or implement the apparatus of any of Examples 1-22.

[0171] The flowchart and block diagram of this paper illustrate the architecture, functionality and operation of the possible implementation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each frame in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or the part of the code includes one or more executable instructions for realizing the specified logical function. It should also be noted that each frame in the block diagram and / or the flowchart and the combination of the frames in the block diagram and / or the flowchart can be realized by a special hardware-based system or a special hardware and computer instruction combination that performs a specified function or action. These computer program instructions can also be stored in a computer-readable medium, and the computer program instructions can guide a computer or other programmable data processing device to work in a particular way, so that the instructions stored in the computer-readable medium produce the articles of manufacture, and the instructions include the instructions for realizing the function / action specified in one or more frames of the flowchart and / or the block diagram.

[0172] The systems and methods described herein use new data processing platforms, methods, systems, and algorithms to provide flexible and scalable data warehouses. In some embodiments, the described systems and methods utilize cloud infrastructure that supports cloud-based storage resources, computing resources, and the like. The example cloud-based storage resources provide large amounts of storage capacity on demand at low cost. In addition, these cloud-based storage resources can be fault-tolerant and highly scalable, which may be costly to implement in private data storage device systems. The example cloud-based computing resources can be obtained on demand and can be priced based on the actual usage level of the resources. Typically, cloud infrastructure can be dynamically deployed, reconfigured, and deactivated in a rapid manner.

[0173] In the described systems and methods, the data storage device system utilizes a relational database based on SQL (Structured Query Language). However, these systems and methods are applicable to any type of database using any data storage architecture and using any language to store and retrieve data in the database. The systems and methods described herein can also provide a multi-tenant system that supports isolation of computing resources and data between different customers / clients and between different users within the same customer / client.

[0174] Various technologies or some aspects or parts thereof may be in the form of program codes (i.e., instructions) contained in tangible media (such as floppy disks, CD-ROMs, hard disk drives, non-transitory computer-readable storage media, or any other machine-readable storage media), wherein when the program code is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing various technologies. In the case of executing program code on a programmable computer, a computing device may include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. Volatile and non-volatile memory and / or storage elements may be RAM, EPROM, flash drives, optical drives, magnetic hard disk drives, or other media for storing electronic data. One or more programs that may implement or utilize various technologies described herein may use application programming interfaces (APIs), reusable controls, etc. Such programs may be implemented in high-level stylized programming languages ​​or object-oriented programming languages ​​to communicate with a computer system. However, if necessary, programs may be implemented in assembly or machine language. In any case, the language may be a compiled language or an interpreted language, and may be combined with a hardware implementation.

[0175] It should be understood that many of the functional units described in this specification can be implemented as one or more components, and these terms are used to more specifically emphasize the independence of their implementation. For example, a component can be implemented as a hardware circuit that includes a customized very large scale integration (VLSI) circuit or gate array, a ready-made semiconductor (such as a logic chip, a transistor), or other discrete components. Components can also be implemented in programmable hardware devices (such as field programmable gate arrays, programmable array logic, programmable logic devices, etc.).

[0176] Parts can also be implemented with software, to be performed by various types of processors. The identification parts of executable code can for example include one or more physical blocks or logic blocks of computer instructions, which can for example be organized as objects, processes, or functions. However, the executable file of the identification parts does not need to be physically located together, but can include different instructions stored in different locations, which, when these instructions are combined logically, have just constituted the parts and realized the described purpose of the parts.

[0177] In fact, the parts of executable code can be single instructions, or many instructions, and can even be distributed on several different code segments, distributed between different programs and distributed on several memory devices. Similarly, operating data may be identified and illustrated in the parts herein, and can be embodied in any suitable form, and organized in the data structure of any suitable type. Operating data can be collected with a single data set, or can be distributed in different locations, including being distributed on different storage devices, and can be present in a system or network at least in part as an electronic signal. Parts can be passive or active, including agents that can be operated to perform desired functions.

[0178] References throughout this specification to "example" mean that a particular feature, structure, or characteristic described in conjunction with the example is included in at least one embodiment of the present disclosure. Therefore, the phrase "in an example" appearing in various places throughout this specification does not necessarily refer to the same embodiment.

[0179] As used herein, for convenience, multiple projects, structural elements, constituent elements, and / or materials may be presented in a common list. However, these lists should be interpreted as each member in the list being individually identified as an independent and unique member. Therefore, in the absence of contrary instructions, any individual member in such a list should not be interpreted as the de facto equivalent of any other member in the same list simply based on its presentation in a common group. In addition, various embodiments and examples of the present disclosure may be cited together with the alternatives of its various parts in this article. It should be understood that such embodiments, examples, and alternatives should not be interpreted as de facto equivalents to each other, but should be considered as independent and autonomous presentations of the present disclosure.

[0180] Although the foregoing has been described in detail for the sake of clarity, it is apparent that certain changes and modifications may be made without departing from the principles thereof. It should be noted that there are many alternative ways to implement the processes and devices described herein. Therefore, the present embodiments are considered to be illustrative rather than restrictive.

[0181] Those skilled in the art will appreciate that many changes may be made to the details of the above-described embodiments without departing from the basic principles of the present disclosure. Therefore, the scope of the present invention should be determined solely by the appended claims.

Claims

1. A method comprising: storing a current version of a database table in a current version set of one or more table data files, the database table comprising rows and columns; storing table metadata for a current version of the database table in a current version set of one or more table metadata files in an immutable storage device, the immutable storage device comprising a data storage device that does not update data in place; detecting a modification to a current version of the database table, the modification generating a new version of the database table, the new version of the database table being stored in a new version set of one or more table data files, the new version set of the one or more table data files being different from a current version set of the one or more table data files; and In response to detecting a modification to the current version of the database table, a new version set of one or more table metadata files is generated and stored in the immutable storage device, wherein the new version set of one or more table metadata files includes table metadata of a new version of the database table.

2. The method according to claim 1, wherein: Both the current set of versions of the one or more table data files and the new set of versions of the one or more table data files are stored in the immutable storage.

3. The method according to claim 1, wherein: The cloud storage service includes the immutable storage device.

4. The method according to claim 1, wherein: Both the table metadata of the current version of the database table and the table metadata of the new version of the database table are stored in one or more table metadata file sets of each in a column-by-column format, such that the table metadata related to any given column of the database table is stored in the table metadata in one or more consecutive blocks.

5. The method according to claim 4, further comprising: storing a hash value of table metadata for a given column of the database table; as well as The stored hash value is compared to the calculated hash value to determine whether table metadata for a given column of the database table has been changed.

6. The method according to claim 1 also includes storing metadata location data in a variable storage device, wherein the metadata location data indicates the storage location of one or both of the current version set of the one or more table metadata files and the new version set of the one or more table metadata files, and the variable storage device includes a data storage device in which data can be updated in place.

7. The method according to claim 1, wherein: The table metadata for the new version of the database table indicates: any one or more new table data files generated by the modification, the one or more new table data files being in a new version set of the one or more table data files but not in a current version set of the one or more table data files; as well as Any one or more deleted table data files generated by the modification are not in the new version set of the one or more table data files after the one or more deleted table data files are already in the current version set of the one or more table data files.

8. The method according to claim 1, further comprising: When processing the first query, caching one or more table metadata files in one or both of a current version set of the one or more table metadata files and a new version set of the one or more table metadata files; receiving a subsequent query directed to the database table, and in processing the subsequent query, responsively downloading any uncached table metadata files in a scan set of table metadata files for the subsequent query, the downloading comprising downloading from the immutable storage device; as well as The subsequent queries are processed using the downloaded table metadata file.

9. The method according to claim 8, wherein: The downloading includes downloading a plurality of table metadata files from the immutable storage device in parallel.

10. The method according to claim 9, further comprising: A first table metadata file is read before a second table metadata file is completely downloaded, the plurality of table metadata files including both the first table metadata file and the second table metadata file.

Images