A cloud-native big data analytics engine

Abstract

This invention discloses a cloud-native big data analytics engine, specifically relating to the field of big data analytics technology. The engine comprises the following: dividing an OLAP database into different partitions according to data distribution, with each partition called a Shard. Each Shard stores its data files in S3 object storage. This invention proposes a novel design and implementation of an OLAP analytical database, suitable for enterprises deploying on public cloud infrastructure, providing a cost-effective and high-speed OLAP analytical database solution. It is a crucial component of enterprise cloud-native infrastructure. Furthermore, this invention is not limited to public cloud deployments; it can also provide a fast, cost-effective, and high-performance OLAP analytics engine in private data centers. Therefore, enterprise applications using this invention as a foundational component can easily switch between different public and private clouds.

Description

Technical Field

[0001] This invention relates to the field of big data analytics technology, and more specifically to a cloud-native big data analytics engine. Background Technology

[0002] OLAP analytical databases are one of the cornerstones of public cloud services, enabling analysts to quickly, consistently, and interactively gain insights into data from various perspectives to achieve a deep understanding of the data. Public cloud OLAP databases fall into several categories: migrating open-source OLAP databases to the public cloud to provide DBaaS (Database as a Service); and public cloud providers developing their own OLAP databases specifically for cloud users. When deploying applications on the public cloud, different enterprises have two different approaches to choosing an OLAP database: one is to directly adopt one of the two types of managed OLAP database services mentioned above; the other is to purchase public cloud infrastructure, including compute and storage, and then deploy and maintain the open-source OLAP database themselves.

[0003] Existing technologies have the following shortcomings: Enterprises face various difficult choices when migrating to cloud-native architectures. This is not only due to the complexity and limitations of OLAP technology itself, but also because the resources available for purchase in public clouds have unique characteristics compared to private deployments. Public clouds offer basic computing resources including virtual machines and physical hosts, the latter being expensive, while the former typically provides only a small amount of local storage. Moreover, large-scale virtual machines in public clouds usually have a large amount of local storage mounted on a single machine, making it difficult to utilize more parallel computing resources. Therefore, choosing open-source OLAP hosted in public clouds—whether row-based, column-based, or index-based—is neither cheap nor high-performance; it cannot achieve high cost-effectiveness in storage or optimal performance. Summary of the Invention

[0004] To address this, the present invention provides a cloud-native big data analytics engine. Through the design and implementation of a novel OLAP analytical database, it is suitable for enterprises deploying on public cloud infrastructure, providing a cost-effective and high-speed OLAP analytical database solution and serving as a crucial component of enterprise cloud-native infrastructure. Furthermore, the present invention is not limited to public cloud deployments; it can also provide a fast, cost-effective, and high-performance OLAP analytics engine in private data centers. Therefore, enterprise applications using this invention as a foundational component can easily switch between different public and private clouds, resolving the issues of high cost and low performance associated with choosing publicly hosted open-source OLAP in existing technologies.

[0005] To achieve the above objectives, embodiments of the present invention provide the following technical solution: a cloud-native big data analytics engine, comprising the following:

[0006] S1. Divide the OLAP database into different partitions according to the distribution of data, and call each partition a Shard. Each Shard stores its data files in S3 object storage.

[0007] S2. For each inserted record, store the record in row storage format and save it to the S3 object storage; generate a corresponding column storage for each column of the record and save it to the S3 storage; assign an auto-incrementing integer ID to each record, and then use Bitmap indexing (BSI) to build the corresponding inverted index for the column of the record.

[0008] S3. Key-Value interface storage is used for row storage and Bitmap inverted index: the open-source Key-Value embedded engine Pebble is used to provide the Key-Value interface; the underlying Pebble is an SST file, which is compressed at different levels; a separate batch interface is implemented for column storage: a simple block index is built in memory: the maximum and minimum values ​​of each fixed number of columns are recorded, thereby avoiding invalid IO during scanning;

[0009] S4 provides a global WAL log service that is shared and accessed by all Shards. When inserting data, the data is first inserted into the global WAL and then inserted into the different Shards.

[0010] S5. Provides comprehensive SQL support: The SQL execution layer is built based on relational algebra rather than on Plan, and can directly determine based on attributes: if attribute a has an inverted index, then the index plan is executed; otherwise, columnar storage is used, and a full table scan is performed followed by filtering.

[0011] Furthermore, the inverted index contains a set of Bitmaps, which are converted into binary representations based on the selected column values ​​and vertically sliced ​​into Bitmaps. For columns containing cardinality strings, no inverted index is constructed.

[0012] Furthermore, all storage within the Shard, including Pebble storage with Key-Value interfaces and dedicated columnar storage, will have WAL logging disabled.

[0013] Furthermore, each shard is mounted to the file cache of the block storage, and different caching mechanisms are provided for the Pebble storage and the column storage's proprietary format, which are relied upon by row storage and Bitmap inverted index. For Pebble storage, the cached objects are based on the underlying SST files of Pebble. Specifically, SST files in Level 0 and Level 1 are first stored in the file cache and then pushed to the S3 object storage. For column storage, a conventional LRU caching mechanism is used, based on column storage file units.

[0014] Furthermore, the global WAL log service is selected from the open-source Pulsar, which includes a block storage and a corresponding S3 object storage at the underlying level.

[0015] Furthermore, each Shard starts only one instance by default, and its working order is as follows: the new virtual machine loads the object files stored in a certain Shard from the shared S3 object storage, and continues to consume data from the global WAL log service to ensure the consistency of subsequent data.

[0016] Furthermore, the cloud-native big data analytics engine supports schema-free data import, accepting any JSON data as input. For basic JSON types, it will be interpreted as basic SQL types String, Number, and Boolean, while null will be treated as NULL in SQL.

[0017] Furthermore, the native big data analytics engine serves both public and private cloud environments.

[0018] The present invention has the following advantages:

[0019] This invention is an OLAP analytical database based on public cloud S3 object storage. To balance cost and high performance, this invention uses multiple data storage formats, including row storage, column storage, and inverted indexes. This allows different formats to serve different types of queries, fully leveraging the different advantages of row storage, column storage, and indexes to serve different types of queries, minimizing the user's selection cost, and providing users with high-performance query services.

[0020] This invention solves the problem that ordinary inverted indexes cannot serve full SQL capabilities. For numeric types, including floating-point numbers, inverted indexes are still used to provide high-concurrency SQL filtering and SQL aggregation capabilities, and the inverted indexes work on top of S3 object storage.

[0021] This invention provides a schema-free data insertion method on top of public cloud object storage to address diverse business scenarios. By providing schema-free data insertion capabilities, users do not need to predefine database fields and types when inserting data, which greatly simplifies the data insertion process. Furthermore, no modifications to the database schema are required for complex and ever-changing business scenarios. This invention not only provides deployment capabilities on top of public cloud object storage but also provides the same query and analysis services in private environments. This is due to the encapsulation of the existing Multi-Raft mechanism, which allows it to run as a separate database and simultaneously manage both conventional key-value engines and customized columnar storage. Moreover, this invention provides a unified encapsulation of the mechanisms for managing local storage and S3 object storage based on the Multi-Raft mechanism. Attached Figure Description

[0022] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0023] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0024] Figure 1 The basic framework diagram provided for this invention;

[0025] Figure 2 A binary diagram of the BSI index provided by this invention;

[0026] Figure 3 This is a schematic diagram of the cloud environment provided by the present invention. Detailed Implementation

[0027] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Refer to the instruction manual appendix Figure 1-3 The present invention provides a cloud-native big data analytics engine, comprising the following:

[0029] OLAP databases are divided into different partitions based on data distribution, and each partition is called a Shard. Each Shard stores data files in S3 object storage. At the same time, to overcome the insufficient throughput of the S3 file system, each Shard itself mounts a storage block to serve as a file cache for the S3 storage.

[0030] For each inserted record, the present invention performs the following processing:

[0031] 1. Store the records in row-based format and save them to S3 object storage.

[0032] 2. Generate a corresponding columnar storage for each column of the record and save it to S3 storage.

[0033] 3. Assign an auto-incrementing integer ID to each record, and then generate corresponding inverted indexes for each column of the record. Specifically, the inverted indexes are built using Bitmap technology. The Bitmap indexes are also stored in S3 object storage.

[0034] Therefore, in this invention, all data exists in three formats: row-based, column-based, and Bitmap inverted index. Although the data size increases at least three times compared to the original records, because it is stored in S3 object storage, the cost is less than 1 / 10 of local disk or block storage. Therefore, the overall overhead is not only not increased compared to other OLAP databases, but is significantly reduced. In the SQL query plan, as long as the query columns have a Bitmap inverted index, the SQL execution will be completed using the Bitmap inverted index.

[0035] Not all columns can have inverted indexes created. Typically, inverted indexes can only be created for enumerated strings, as is the case with Elasticsearch. For numeric types, the latter uses the inefficient BKDTree, which has poor query filtering performance and cannot satisfy aggregate queries on numeric columns. This invention uses BSI technology to create a Bitmap inverted index. The inverted index contains a set of Bitmaps, which are converted to binary representations of the selected column values ​​and then vertically sliced ​​into Bitmaps, such as... Figure 2 As shown. Figure 2 For example, only 15 Bitmaps are needed to represent values ​​from 0 to 30000, so row numbers can be easily set in these Bitmaps, thus creating a Bitmap inverted index for integer fields as well. For floating-point number fields, this invention also uses BSI technology, only requiring consideration of cardinality and mantissa separately. BSI indexes solve the drawback of inverted indexes in creating aggregate queries and high-speed range filtering for numeric fields. However, for string columns with high cardinality, creating a Bitmap index offers no advantage; a typical example is an MD5-encrypted user string ID. This invention can choose not to build an inverted index for such scenarios.

[0036] In this invention, row-based and column-based storage can be disabled through configuration. For OLAP databases, row-based storage is only used in rare cases, such as random point queries. This type of query is more common in OLTP databases. Disabling row-based storage as needed can further conserve resources. Column-based storage is also rarely used, typically only when the columns involved in the query lack inverted indexes, or when the analytical query on the columns includes LIKE types requiring extensive scans.

[0037] In this invention, the row storage and Bitmap inverted index are stored using a Key-Value interface. The key of the inverted index is the specific value of the column, and the value is a list of row numbers in which that specific value exists. Since Bitmap itself is a sparse data structure, this invention uses the Roaring Bitmap format to compress the Bitmap inverted index, reducing IO overhead. To avoid the overhead of continuous updates to the inverted index, this invention first builds the inverted index in memory and only writes it to storage after the memory bucket is full. This invention uses the open-source Key-Value embedded engine Pebble to provide the Key-Value interface. Pebble's underlying structure is an SST file, which is compressed at different levels. Its working mechanism is similar to the popular RocksDB, but it does not require WAL logs and multi-row transaction guarantees, and has a throughput several times higher than RocksDB.

[0038] In this invention, columnar storage does not rely on a Key-Value interface, as this would significantly reduce the throughput of columnar storage record scanning. Therefore, this invention implements a separate batch interface for columnar storage, characterized by direct storage of data for each column of each record without any serialization overhead. A simple block index is built in memory: recording the maximum and minimum values ​​of a fixed number of columns, thereby avoiding unnecessary I / O during scanning. The columnar storage format of this invention borrows from the single-machine storage engine format of the open-source OLAP engine ClickHouse, as the latter is currently the fastest open-source OLAP database.

[0039] In this invention, each shard is mounted to a file cache in block storage. Different caching mechanisms are provided for the Pebble storage (which relies on the row-based storage and Bitmap inverted index) and the column-based storage's proprietary format. For Pebble storage, cached objects are based on the underlying Pebble SST files. Specifically, Level 0 and Level 1 SST files are preferentially stored in the file cache before being pushed to the S3 object storage. For column-based storage, a conventional LRU caching mechanism is used, based on column-based file units.

[0040] This invention disables the Write-Ahead Log (WAL) function for all storage within a shard, including Pebble storage with a Key-Value interface and dedicated columnar storage, in order to reduce I / O overhead. This invention also provides a global WAL log service that can be shared and accessed by all shards. When inserting data, the data is first inserted into the global WAL, and then inserted into the respective shards. This way, even if the virtual machine residing in a shard fails, another virtual machine can be quickly rebuilt, and the data can be recovered and reconstructed from the global WAL.

[0041] The global Write-Ahead Log (WAL) service was chosen to be the open-source Pulsar. The global WAL service includes an underlying block storage and a corresponding S3 object storage. Pulsar is responsible for migrating cold data from the WAL to S3. The popular Kafka was not chosen because it is not very cloud-native and lacks the ability to automatically migrate hot and cold data.

[0042] In this invention, each Shard starts only one instance by default. However, in some cases, it may be necessary to start additional replica instances for certain Shards. The working order is as follows: the new virtual machine loads the object files stored in a certain Shard from the shared S3 object storage and continues to consume data from the global WAL log service to ensure the consistency of subsequent data.

[0043] A typical scenario for launching additional replica instances is when the system's throughput reaches its limit, requiring an increase in throughput by adding replicas. Therefore, to make this process as fast as possible, this invention uses Kubernetes containers for orchestration. Thus, the Shard instances of this invention actually run within containers, and are not necessarily virtual machines. This invention dynamically requests container resources from the Kubernetes container cloud based on the queried load.

[0044] This invention provides comprehensive SQL support. The SQL execution layer is built on relational algebra rather than on a plan.

[0045] This invention directly determines based on attributes: if attribute a has an inverted index, then the index plan is executed; otherwise, columnar storage is used, and a full table scan is performed followed by filtering.

[0046] This invention can be configured to support schema-free data import, accepting any JSON data as input. Basic JSON types are interpreted as basic SQL types: String, Number, and Boolean. Null values ​​are treated as SQL NULL. JSON Objects are treated as SQL sub-relations, and Arrays exist as arrays, but the type of each element in the array can be different. In this invention, the types of values ​​for a specific attribute can be different, but each value has a defined type. Furthermore, it supports the special type of sub-relations. For example, for attribute 'a', if it accepts two values ​​{"a": "B"} and {"c": 1, "a": "c"}, then there exists a sub-relation for attribute 'a' with attributes 'a' and 'c', containing two tuples.

[0047] While this invention primarily serves public cloud environments, it can also be deployed and used in private cloud environments. This includes two scenarios: If the private cloud provides an S3-compatible object storage interface, such as the Ceph distributed file system, then this invention can be deployed directly. If the private cloud only provides physical machines, then this invention will rely on its own storage abstraction library to manage the underlying storage, specifically defined as follows... Figure 3 As shown. Figure 3It is a distributed storage repository based on the Multi-Raft multi-group strong consistency protocol. Some newer databases, such as TiDB, use similar mechanisms to provide underlying storage. This invention abstracts the Multi-Raft mechanism, separating it from the underlying storage engine and decoupling it from upper-layer applications. Therefore, it is an embedded distributed storage management tool rather than a standalone process. Due to the compute and storage separation architecture of this invention, all the aforementioned designs can run on the Multi-Raft framework in a private environment. The Multi-Raft framework manages Pebble's Key-Value storage engine and also manages the proprietary columnar storage engine, providing multi-replica mechanisms and automatic load balancing for these storage engines.

[0048] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A cloud-native big data analytics engine, characterized in that: The application comprises the following contents: S1, dividing the OLAP database into different partitions according to the distribution of data, and calling each partition as a Shard, and each Shard saves data files in the S3 object storage; S2, for each inserted record, storing the record as a row storage format and saving it to the S3 object storage; generating a corresponding column storage for each column of the record and saving it to the S3 storage; assigning a self-incrementing integer ID to each record, and then using the Bitmap bitmap technology (BSI) to build a corresponding inverted index for the columns of the record; S3, using a Key Value interface to store the row storage and the Bitmap inverted index at the bottom: using the open-source Key Value embedded engine Pebble to provide the Key Value interface; the Pebble is an SST file at the bottom, and corresponding compression will be performed on different levels; a separate batch interface is implemented for the column storage: a simple block index is constructed in the memory: the maximum and minimum values of each fixed number of columns are recorded, so as to avoid invalid IO during scanning; S4, providing a global WAL log service for all Shards to share, and when inserting data, the data is first inserted into the global WAL, and then inserted into different Shards; S5, providing comprehensive and sufficient SQL support: the execution layer of SQL is based on relational algebra rather than Plan, and can be directly judged according to the attribute: assuming that the attribute a exists inverted index, the index plan is executed, otherwise the column storage is used, and the full table scan is filtered. 2.The cloud-native big data analysis engine of claim 1, wherein: The storage in all Shards, including the Pebble storage of the Key Value interface and the column storage, all close the WAL log function. 3.The cloud-native big data analysis engine of claim 1, wherein: Each Shard is mounted to the file cache of the block storage, and different cache mechanisms are provided for the Pebble and the column storage; for the Pebble storage, the cached objects are units of SST files at the bottom of the Pebble; specifically, the SST files at Level 0 and Level 1 are preferentially stored in the file cache, and then pushed to the S3 object storage, and for the column storage, the regular LRU cache mechanism is used in units of column storage files. 4.The cloud-native big data analysis engine of claim 1, wherein: The global WAL log service is selected as the open-source Pulsar, and the global WAL log service comprises a block storage and a matching S3 object storage at the bottom. 5.The cloud-native big data analysis engine of claim 1, wherein: Each Shard starts only one instance by default, and the working order is as follows: a new virtual machine loads the object files stored in a certain Shard from the shared S3 object storage, and continues to consume data from the global WAL log service to ensure the consistency of subsequent data. 6.The cloud-native big data analytics engine of claim 1, wherein: The cloud-native big data analysis engine supports Schema Free data import, at this time, arbitrary JSON data is accepted as input, and for basic JSON types, it is interpreted as basic SQL types String, Number, Boolean, and for null, it is treated as SQL NULL.

7. The cloud-native big data analytics engine of claim 1, wherein: The native big data analysis engine serves public cloud environment and private cloud environment.

Images