Storage engine for hybrid data processing

Abstract

This disclosure describes storage techniques for hybrid transactional and analytical processing. Data captured by a first processing engine can be received. The first processing engine can be configured to perform online transactional processing. Multiple copies of a logical log generated based on the data can be distributed to an incremental repository by applying a consensus protocol on the multiple copies. Data in the incremental repository is stored in a row format and is visible to queries of online analytical processing performed by a second processing engine. Data can be flushed from the incremental repository to a base repository based on one or more predetermined rules. Data in the base repository is stored in a columnar format and is accessible by the second processing engine.

Description

Background Technology

[0001] Data processing refers to the process of performing specific operations on a set of data or a database. A database is an organized collection of facts and information, such as records about inventory, customers, etc. Various forms of data processing exist and serve different applications in a business environment. As databases are increasingly used to store large amounts of complex data, improvements in data processing and storage may be necessary. Attached Figure Description

[0002] The following detailed description will be better understood when read in conjunction with the accompanying drawings. For illustrative purposes, exemplary embodiments of various aspects of this disclosure are illustrated in the drawings; however, this disclosure is not limited to the specific methods and means disclosed.

[0003] Figure 1 An example system including cloud services is shown.

[0004] Figure 2 An example system including more than one cloud service is shown.

[0005] Figure 3 An example system for hybrid transactional and analytical processing is shown.

[0006] Figure 4 An example storage engine for hybrid transactional and analytic processing is shown.

[0007] Figure 5 An example logical log is shown.

[0008] Figure 6 An example data storage architecture for the Delta Store is shown.

[0009] Figure 7 An example data storage architecture for the base store is shown.

[0010] Figure 8 An example process for a storage engine used for hybrid transactional and analytic processing is shown.

[0011] Figure 9 An example process for a storage engine used for hybrid transactional and analytic processing is shown.

[0012] Figure 10 An example process for a storage engine used for hybrid transactional and analytic processing is shown.

[0013] Figure 11 An example computing device is shown that can be used to perform any of the techniques disclosed herein. Detailed Implementation

[0014] Online Analytical Processing (OLAP) systems enable users to interactively analyze multidimensional data from multiple perspectives. Multidimensional data includes datasets with three or more dimensions. OLAP systems allow users to analyze information from multiple database systems simultaneously. OLAP systems enable analysts to extract and view data, such as business data, from different angles. Analysts often need to group, aggregate, and join data. These OLAP operations in data mining are resource-intensive. With OLAP, data can be pre-computed and pre-aggregated, making analysis faster. However, traditional OLAP systems typically load large amounts of data in batches periodically. This can lead to OLAP systems suffering from stale data.

[0015] OLAP is typically contrasted with OLTP (Online Transaction Processing) systems. OLTP systems capture, store, and process data from transactions. A general characteristic of OLTP systems is the large number of relatively simple queries used to process transactions rather than for business intelligence or reporting purposes. Traditional OLTP systems support Data Manipulation Language (DML), a computer programming language used to add (insert), delete, and modify (update) data in a database. Traditional OLTP systems also efficiently support point lookup queries.

[0016] There are significant differences between OLAP and OLTP systems. For example, OLTP systems typically lack massively parallel query engines (such as those found in OLAP systems) to support efficient and complex query processing of large amounts of data. As another example, OLAP systems are primarily optimized for read-only operations and may not support other types of queries, while OLTP systems handle all types of queries (read, insert, update, and delete). As yet another example, OLTP systems are associated with short atomic transactions, while OLAP systems allow for more flexible distribution patterns and higher scalability, but with increased latency and no guaranteed upper limit on processing time.

[0017] Many mainstream data platforms / systems focus on only one of these workloads (e.g., OLAP or OLTP). However, many scenarios (e.g., business scenarios) require both complex OLAP-style analysis of newly imported data and the transaction support and strong data consistency provided by OLTP systems. Traditionally, in data warehouses, data is extracted from OLTP systems, transformed during Extract-Transform-Load (ETL), and then loaded into OLAP systems. The ETL process introduces significant latency (e.g., tens of minutes to hours) between transactions and data visibility into OLAP systems.

[0018] Hybrid Transactional / Analytical Processing (HTAP) systems are best suited for such scenarios. HTAP systems are suitable for business scenarios that require performing complex OLAP-style analysis on newly imported data. Furthermore, HTAP systems are ideal for customers who want transactional support and high data freshness in analytical processing. HTAP systems offer several unique advantages. In an HTAP system, OLAP and OLTP workloads are unified within a single system. By integrating OLAP and OLTP workloads into a single system, the complexity and cost of deployment and maintenance are significantly reduced. Such versatile systems can significantly reduce staleness in query results (which is typically introduced by time-consuming and expensive ETL processes from the operational database to the data warehouse). These systems also have the capability to perform complex analysis on real-time data, thus addressing modern business models that often require effective responses to fleeting opportunities that must be utilized in real time.

[0019] However, many existing HTAP systems have drawbacks. Many existing HTAP systems use offline ETL to ingest data into columnar file format. This offline ETL process often suffers from high latency between OLTP data and OLAP copies. Other HTAP systems maintain row-formatted data for OLTP and column-formatted data for OLAP within a single system. However, such HTAP systems may lack high data freshness or the technology for efficient analytical processing within the HTAP storage engine. Therefore, an HTAP system that addresses these shortcomings is needed.

[0020] This paper describes an HTAP system capable of handling both transactional (OLTP) and complex analytical (OLAP) workloads. Unlike existing HTAP systems, the HTAP system described here utilizes a large-scale real-time analytics architecture that facilitates fresh data changes and strong data consistency. The improved HTAP system is configured for fast DML (Data Manipulation Language) query processing via a row repository in the OLTP engine. The improved HTAP system is further configured for efficient distributed query processing of complex OLAP queries, including join and aggregation operations, within the OLAP engine. Data changes from the OLTP engine are continuously applied to a distributed incremental repository in memory, and the improved HTAP system allows the OLAP engine to query the latest OLTP data changes immediately (or almost immediately).

[0021] The HTAP system described herein includes a storage engine. The storage engine can be decoupled from the OLAP engine and / or OLTP engine. Due to the architecture of one or more separate compute and storage engines, the HTAP storage engine can be configured to support efficient analytical processing and high data freshness. High data freshness may indicate that transactional data from an OLTP database is immediately or nearly immediately visible to an OLAP database. Many features of the HTAP system described herein contribute to efficient analytical processing and high data freshness, including, but not limited to, concurrent logs applied to an in-memory incremental repository, predicate / aggregate pushdown to the storage engine, an efficient columnar base repository for analytical query processing, cost-based optimization of deletion filtering, efficient memory management and garbage collection using site vectors, background flushing / compacting, etc.

[0022] Improved HTAP systems can feature a flexible, modular design. The system's main components, such as the OLTP query engine, OLAP query engine, and / or underlying storage, are modular and decoupled. Therefore, components can be easily changed without modifying the entire architecture. For example, the HTAP storage engine can be decoupled from other components of the system and replaced with a different storage engine. The modular design also makes scaling outwards for each component much easier.

[0023] Many different systems or entities can utilize HTAP systems, such as the improved HTAP system described above. Figure 1 The illustration shows an example system 100 that includes an HTAP system. System 100 may include a cloud network 102 and multiple client devices 104a-d. The cloud network 102 and the multiple client devices 104a-d may communicate with each other via one or more networks 120.

[0024] Cloud network 102 may be located in a data center, such as a single location, or distributed across different geographical locations (e.g., in several locations). Cloud network 102 may provide services via one or more networks 120. Network 120 includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, etc. Network 120 may include physical links, such as coaxial cable links, twisted-pair cable links, fiber optic links, combinations thereof, etc. Network 120 may include wireless links, such as cellular links, satellite links, Wi-Fi links, etc.

[0025] Cloud network 102 may include multiple computing nodes 118 hosting various services. In an embodiment, node 118 hosts service 112. Service 112 may include content streaming services, such as Internet Protocol (IP) video streaming services. Service 112 may be configured to distribute content via various transport technologies. Service 112 is configured to provide content such as video, audio, text data, combinations thereof, etc. Content may include content streams (e.g., video streams, audio streams, information streams), content files (e.g., video files, audio files, text files), and / or other data. Content may be stored in a database. For example, service 112 may include video sharing services, video hosting platforms, content distribution platforms, collaborative gaming platforms, etc. In addition to, or in lieu of, content streaming services, service 112 may include any other type of service.

[0026] In this embodiment, service 112 may be provided to client device 104 via network 120. If service 112 is a content streaming service, content may be output to different client devices 104 via network 120. Content may be streamed to client devices 104. The content stream may be a short video stream received from service 112. Multiple client devices 104 may be configured to access content from service 112. In this embodiment, client device 104 may include an application. The application outputs (e.g., displays, renders, presents) content to the user associated with client device 104. The content may include video, audio, comments, text data, etc.

[0027] In this embodiment, a user can use an application on client device 104 to create content and upload short videos to cloud network 102. Client device 104 can access an interface of the application. This interface may include input elements. For example, the input elements may be configured to allow the user to create content. To create content, the user can grant the application permission to access an image capture device, such as the camera or microphone of client device 104. After the user has created content, the user can use the application to upload the content to cloud network 102 and / or save the content locally to user device 104. Service 112 may store the uploaded content and any metadata associated with that content in one or more databases.

[0028] Multiple client devices 104 can include any type of computing device, such as mobile devices, tablets, laptops, desktop computers, smart TVs or other smart devices (e.g., smartwatches, smart speakers, smart glasses, smart helmets), gaming devices, set-top boxes, digital streaming devices, robots, etc. Multiple client devices 104 can be associated with one or more users. A single user can use one or more of the multiple client devices 104 to access the cloud network 102. Multiple client devices 104 can travel to multiple locations and use different networks to access the cloud network 102.

[0029] Multiple compute nodes 118 can handle tasks associated with service 112. Multiple compute nodes 118 can be implemented as one or more compute devices, one or more processors, one or more virtual compute instances, or combinations thereof. Multiple compute nodes 118 can be implemented by one or more compute devices. One or more compute devices can include virtualized compute instances. Virtualized compute instances can include virtual machines, such as emulations of computer systems, operating systems, servers, etc. Virtual machines can be loaded by compute devices based on virtual images and / or other data defining specific software (e.g., operating systems, dedicated applications, servers) used for emulation. As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more compute devices. A hypervisor can be implemented to manage the use of different virtual machines on the same compute device.

[0030] In this embodiment, service 112 includes HTAP system 110. HTAP system 110 may include multiple different components (e.g., subsystems). For example, HTAP system 110 may include one or more of a transactional OLTP engine, an analytical OLAP engine, underlying decoupled storage, metadata services, and / or intelligent agents.

[0031] HTAP system 110 can have an architecture that supports heterogeneous query engines. This architecture can handle both transactional OLTP workloads and complex analytical (OLAP) workloads. The architecture can follow a modular design, and its main components can be fully decoupled, providing flexibility and easy scalability. For example, components of HTAP system 110 can be easily changed into similar, established subsystems. This architecture can eliminate interference between OLTP and OLAP workloads by having separate query processing engines and different data replicas.

[0032] HTAP system 110 can store user data in different formats for OLTP and OLAP engines. For example, in the storage subsystem, HTAP system 110 can store user data in row format for OLTP engines and column format for OLAP engines for efficient query processing. This architecture can have a single point of truth for metadata and can use a separate metadata service to provide up-to-date metadata to certain components of HTAP system 110. The architecture of HTAP system 110 can include an intelligent agent that dispatches queries to OLTP and OLAP subsystems based on the nature of the query (thus hiding internal details from users / clients). Users / clients can utilize HTAP system 110 with a single unified interface. For example, users / clients might be able to utilize HTAP system 110 with an interface for client device 104. This architecture can support various APIs (e.g., ANSI SQL, JDBC, ODBC, etc.) based on user needs.

[0033] The architecture of the HTAP system 110 is capable of handling large-scale data. This is a result of the fact that the computing and storage components in the HTAP system 110 can be decoupled. Because it is not assumed that the data will fit in memory, a decoupled storage system capable of persistently storing large amounts of data is utilized. The computing and storage resources in the HTAP system 110 can also be scaled out and therefore can flexibly handle large amounts of data and large-scale (OLTP and OLAP) workloads.

[0034] The architecture of HTAP System 110 enables efficient and real-time data processing. DML queries can be efficiently processed by the OLTP engine and written efficiently to the underlying storage in row format. The architecture of HTAP System 110 can include an OLAP query engine with distributed query processing capabilities (high parallelism, better resource utilization) to efficiently handle complex OLAP queries, including joins and aggregations. Because the architecture of HTAP System 110 facilitates storing different copies of data on both the OLTP and OLAP sides, the interference between OLTP and OLAP workloads is minimized, enabling efficient and real-time processing. OLTP and OLAP data formats can be optimized individually to suit their workloads. A single source of data change can exist through HTAP System 110 (from the OLTP side), simplifying the consistency model and concurrent processing across OLTP and OLAP components.

[0035] The architecture of HTAP System 110 provides fresh / real-time data changes for OLAP queries. Logical logs of DML operations are propagated from the OLTP component to the OLAP component immediately upon commit. These logs can be dispatched to distributed partitions and continuously applied to the in-memory incremental repository via typically very fast in-memory operations. Data changes carried by the logical logs are immediately available for OLAP queries when applied to the in-memory incremental repository. The architecture of HTAP System 110 leverages unified versioning across HTAP System 110, ensuring strong data consistency. Like most transactional OLTP database engines, the OLTP component of HTAP System 110 supports snapshot isolation and other (weaker) consistency models.

[0036] although Figure 1 System 100 presents HTAP system 110 as being provided by a single cloud network 102, but the various components / subsystems of HTAP system 110 may alternatively be provided by multiple different cloud networks. Figure 2 An example system 200 is illustrated, comprising an HTAP system with components / subsystems spanning multiple cloud networks. System 200 may include cloud networks 202a-b and multiple client devices 204a-d. Cloud networks 202a-b and multiple client devices 204a-d may communicate with each other via one or more networks 220.

[0037] Each of the cloud networks 202a-b can be similar to the above. Figure 1 The cloud network 102 described herein. Each of the cloud networks 202a-b may be located at a data center, such as a single site, or distributed across different geographical locations (e.g., at several sites). The cloud network 202a-b may provide services (one or more) via one or more networks 220. The cloud network 202a-b includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, etc. The cloud network 202a-b may include physical links, such as coaxial cable links, twisted-pair cable links, fiber optic links, combinations thereof, etc. The cloud network 202a-b may include wireless links, such as cellular links, satellite links, Wi-Fi links, etc.

[0038] Each of cloud networks 202a-b may include multiple compute nodes hosting various services. In one embodiment, there is a node hosting service 212a associated with cloud network 202a, and a node hosting service 212b associated with cloud network 202b. Services 212a-b may include any type of service, such as those referenced above. Figure 1 The content streaming service described.

[0039] Multiple client devices 204 can include any type of computing device, such as mobile devices, tablets, laptops, desktop computers, smart TVs or other smart devices (e.g., smartwatches, smart speakers, smart glasses, smart helmets), gaming devices, set-top boxes, digital streaming devices, robots, etc. Multiple client devices 104 can be associated with one or more users. A single user can use one or more of the multiple client devices 104 to access at least one of the cloud networks 202a-b. Multiple client devices 104 can travel to multiple locations and use different networks to access the cloud networks 202a-b.

[0040] In this embodiment, each service in services 212a-b includes one or more components / subsystems of an HTAP system (e.g., HTAP system 110). HTAP system 110 may include multiple different components (e.g., subsystems). For example, the HTAP system may include one or more of a transactional OLTP engine, an analytical OLAP engine, underlying decoupled storage, metadata services, and / or intelligent agents. Service 212a may include one or more (but not all) components of the HTAP system. Service 212b may include the remaining components of the HTAP system. Although... Figure 2 Two cloud networks and their corresponding services are shown in the document, but it should be understood that any number of cloud networks / services can be used to implement the HTAP system described herein.

[0041] Figure 3 An exemplary architecture 300 for an HTAP system (e.g., HTAP system 110) according to this disclosure is shown. Architecture 300 includes multiple different components (e.g., subsystems). Subsystems include an OLTP engine 303, an OLAP engine 305, a data storage system 306, a metadata service 304, and / or a proxy 302.

[0042] The OLTP engine 303 can receive user data from user devices 301a-n. The OLTP engine 303 can utilize a model that separates the computing engine from the underlying shared / cloud storage. The OLTP engine 303 can provide features such as ACID transaction support, row storage, write-ahead logging (WAL), and log replication. For example, the OLTP engine 303 can capture user data in real-time or near real-time. The OLTP engine 303 can include a data warehouse. The data warehouse can store the data captured by the OLTP engine 303.

[0043] The OLTP engine 303 can be configured to process at least some of the received user data in a specific format (e.g., a row-based format). For example, upon receiving an OLTP query, the OLTP engine 303 can be configured to process at least some of the received user data in response to that query. As an example, and not a limitation, an OLTP query can include queries that retrieve specific data items, queries that filter received data to find specific data items / data item descriptions, and / or queries that filter received data to identify specific subsets of the received data. For example, the query can be a DML query and / or a point lookup query.

[0044] OLTP engine 303 can be configured to be coupled to and / or decoupled from a larger HTAP system 300. If OLTP engine 303 is decoupled from the larger HTAP system 300, then OLTP engine 303 can be replaced with an alternative OLTP engine (e.g., a similar established OLTP engine). For example, OLTP engine 303 could be MySQL, but it can be easily swapped to any other established OLTP engine, such as PostgreSQL.

[0045] User equipment 301a-n can include any type of computing device, such as mobile devices, tablets, laptops, desktop computers, smart TVs or other smart devices (e.g., smartwatches, smart speakers, smart glasses, smart helmets), gaming devices, set-top boxes, digital streaming devices, robots, etc. User equipment 301a-n can be associated with one or more users. A single user can use one or more user equipment 401a-n to access the cloud network including the OLTP engine 303. User equipment 301a-n can travel to multiple locations and use different networks to access the cloud network including the OLTP engine 303.

[0046] OLAP engine 305 can access data captured by OLTP engines (such as OLTP engine 303). The data accessed by OLAP engine 405 can be in a specific format readable by OLAP engine 305 (e.g., a mixed row + column format). OLAP engine 305 can perform analysis on at least a portion of the data in the specific format in response to receiving an OLAP query.

[0047] For example, the OLAP engine 305 can perform analytical operations on at least a portion of the data. OLAP consists of three basic analytical operations: merging (rolling up), drilling down, and slicing and dicing. Merging involves the aggregation of data that can be accumulated and computed across one or more dimensions. For example, all sales offices are aggregated into a sales department or sales division to predict sales trends. In contrast, drilling down is a technique that allows users to browse details. For example, a user can view sales by individual products that make up sales in a particular region. Slicing and dicing are features from which a user can extract (slice) a cube of OLAP data and view (dic) the slices from different perspectives. These viewpoints are sometimes referred to as dimensions (such as viewing the same sales by salesperson, by date, by customer, by product, by region, etc.). The results of the processing can be sent or forwarded to client devices 301a-n.

[0048] OLAP engine 305 can be any OLAP engine capable of efficiently handling complex analytical queries, such as established open-source query engines (e.g., FlinkSQL). OLAP engine 305 can be configured to be coupled to and / or decoupled from a larger HTAP system 300. If OLAP engine 305 is decoupled from the larger HTAP system 300, it can be replaced with an alternative OLAP engine (e.g., a similar established OLAP engine). For example, OLAP engine 305 could be FlinkSQL, but it can be easily swapped to any other established OLAP engine, such as Presto or SparkSQL.

[0049] Storage subsystem 306 can be decoupled from OLTP engine 303 and OLAP engine 305. Data storage system 306 can persistently store user data in one format (e.g., row format) for use by OLTP engines (such as OLTP engine 303), while persistently storing the same user data in a different format (e.g., mixed row + column format) for use by OLAP engines (such as OLAP engine 305). This can significantly reduce interference between OLAP and OLTP workloads. Figure 3 As shown, storage subsystem 306 can be configured to be coupled to and / or decoupled from the larger HTAP system 300. If storage subsystem 306 is decoupled from the larger HTAP system 300, then storage subsystem 306 can be replaced with an alternative storage subsystem (e.g., a similarly established storage subsystem). References will follow. Figure 4 The architecture of storage subsystem 306 will be discussed in more detail.

[0050] Metadata service 304 can be configured to align metadata between OLTP engine 303 and OLAP engine 305. Metadata service 304 can extract metadata from events (e.g., DDL) generated by certain resources (e.g., user input via OLTP engine 303), generate metadata versions, and align them with the order of the DML operations, making them globally available and persistent. Metadata service 304 can generate copies of the metadata versions for high availability. Metadata service 304 can extract metadata (with a version consistent with the same LSN system used for the DML) from DDL received from the OLTP engine, which is persistently stored in a dedicated database and pushed / pulled by FlinkSQL and AP storage servers.

[0051] Metadata service 304 can be configured to be coupled to and / or decoupled from larger HTAP system 300. If metadata service 304 is decoupled from larger HTAP system 300, metadata service 304 can be replaced by an alternative metadata service (e.g., a similar established metadata service).

[0052] Agent 302 can be configured to connect user devices 301a-n to OLTP engine 303 and / or OLAP engine 305 through a common agent layer. Agent 302 can be an intelligent agent. Agent 302 can provide a single, unified API to users / clients (default: ANSI SQL plus some common OLAP extensions), meaning that the underlying system details are transparent to the client if the client connects through the agent. Various types of APIs (e.g., ANSI SQL, JDBC, ODBC, etc.) can be utilized based on user needs. Utilizing a single, unified API reduces the workload for users using the system. Because OLTP engine 303 and OLAP engine 305 each maintain a complete copy of the data, both DML / bulk loading and queries operate on the entire HTAP system 300. Agent 302 can have the ability to automatically dispatch different client requests / queries to different engines (e.g., OLTP engine 303 or OLAP engine 305) based on the nature of the request. For example, complex OLAP queries will be directed to OLAP engine 305, while DML, DDL, and point lookup queries will be directed to OLTP engine 303.

[0053] Agent 302 can be configured to be coupled to and / or decoupled from the larger HTAP system 300. If agent 302 is decoupled from the larger HTAP system 300, agent 302 can be replaced by an alternative public agent (e.g., a similar established agent).

[0054] An HTAP system (e.g., HTAP system 110) may include a data storage component. Figure 4 Showing more details Figure 3 Storage system 306. In data storage system 306, high data freshness is achieved by designing an incremental repository within a distributed memory with multiple replicas and an efficient log distributor that applies an arbitration protocol to the multiple replicas.

[0055] Data storage system 306 can persistently store user data in one format (e.g., row format) for use by the OLTP engine, while simultaneously persistently storing the same user data in a different format (e.g., a mixed row + column format) for use by the OLAP engine. This can significantly reduce interference between OLAP and OLTP workloads. For high availability, multiple copies of the data can be stored for both data formats (e.g., three copies).

[0056] Data storage system 306 can be used as a unified storage layer. However, the architecture of data storage system 306 can be divided into two parts: the TP part and the AP part. Transactional DML and DDL received by the OLTP engine can be presented as physical / redo logs (with information from the underlying storage) and logical logs in log storage 402. These logs can be persistently stored in the TP part of the storage. See below for reference. Figure 5 The exemplary logical log will be discussed in more detail.

[0057] The physical log can then be replicated and distributed to other storage devices by the log distributor 404 and replayed to build data pages. For example, the log distributor 404 can distribute the write-ahead log (WAL) to the row store 406 in shared storage. Data in the pages can be organized in a row format and stored in the row store 406. The OLTP engine can then use the data stored in the row store 406 for simple queries that include point lookup queries.

[0058] Logical logs can also be replicated and distributed to the AP portion of storage by log distributor 404, such as to each partition 407. Each user table in the AP portion of storage can be partitioned based on a partitioning scheme defined when the table is created. Each partition 407 can reside on a physical storage node (e.g., a server), and for high availability, several copies of the data can be maintained in each partition. Each partition 407 can be further divided into an in-memory incremental repository 408 and an on-disk basic repository 410. Logical logs (such as MySQL binary logs) for committed DML transactions can be continuously dispatched from the OLTP engine to the user table partitions on each AP storage node. As an example, updates can be modeled as deletions following inserts. Therefore, in the logical log, there may only be inserts and deletions, without updates.

[0059] Logical logs arriving at each AP storage node can be categorized, persistently stored, and then sequentially applied to the in-memory incremental repository 408 of each partition 407. Once logical logs arrive, they can be sequentially applied to the in-memory incremental repository 408. The latency between the time of data changes within the OLTP engine and the time the data changes appear within the incremental repository is virtually nonexistent (e.g., less than a few hundred milliseconds in most cases). The incremental repository 408 can store data in a row format, sorted by log sequence number (LSN, a sequence number / version indicating the order of data operations). Data within the incremental repository 408 can be periodically flushed to the base repository 410, for example, when its size exceeds a predefined threshold or after a fixed time interval. After a flush, the memory occupied by the flushed data can be collected by the trash can.

[0060] Data in the basic repository 410 can be organized in a columnar format for better performance in analytical workloads and persistently stored / saved on the local file system. Each partition 407 can be stored in a separate directory, and data for different columns can be stored in different files. In each data file, data is categorized by primary key (one or more columns). Furthermore, data blocks can be periodically merged with a compaction process in a background task.

[0061] Data in the base repository 410 may not be updatable in-place, so a deletion bitmap marking deleted rows can be maintained. Because data blocks are not modified once generated, a key-value (KV) system can be used to store the deletion bitmap, which contains information about rows removed from the base repository 410 after flushing from the incremental repository 408.

[0062] As more and more data units are deleted and overlapped during flushing, compressing and rearranging the internal data can benefit storage and potential query performance, thus allowing for periodic compression of AP data units and cleanup of redundant data. Files in the base repository 410 can be versioned, which is the largest LSN in the file at the time of flushing. The LSN may not be stored in the base repository 410, potentially resulting in all files in the base repository 410 having the same version (i.e., the LSN of the last flush). This can offer several advantages. For example, it can save storage space, make deletion more efficient by using a deletion bitmap, and make scans faster without requiring comparison with the LSN. However, not storing the LSN in the base repository 410 may mean that users will not be able to query older versions than the most recently flushed version from the incremental repository 408. In some embodiments, the LSN for the data can be stored in the base repository 410 to support time-based queries (with some cost in terms of storage and scan performance).

[0063] Figure 5 The illustration shows an example logical log 500. To maintain a history of transactions and database server changes since the last storage backup, a database server can generate log records. The database server can store these log records in a logical log, which is a circular file consisting of three or more logical log files. The log is called a logical log because the log records represent logical operations performed by the database server, rather than physical operations.

[0064] As discussed above, transactional DML and DDL received by the OLTP engine can be presented in the log store (e.g., log store 302) as physical / redo logs (with information from the underlying storage) and logical logs, such as logical log 500. Among other data, logical log 500 includes data indicating partition keys, tables, data indicating one or more operations, and data indicating primary keys. For example, column 502 stores data indicating partition keys. Column 504 stores data indicating tables. Column 506 stores data indicating one or more operations. Column 506 stores data indicating primary keys. One or more operations can include any operation, including but not limited to delete, insert, or DDL.

[0065] Logical logs (such as logical log 500) can be replicated and distributed by a log distributor (e.g., log distributor 404) to the AP portion of a storage system (e.g., storage system 306), such as to each partition (e.g., one or more partitions 407). Each user table in the AP portion of storage can be partitioned based on a partitioning scheme defined at table creation time. Each partition can reside on a physical storage node (e.g., a server), and for high availability, several copies of the data can be maintained in each partition. Each partition can be further divided into an in-memory incremental repository (e.g., in-memory incremental repository 408) and disk-based storage (e.g., disk-based storage 410). Logical logs (such as MySQL binary logs) for committed DML transactions can be continuously dispatched from the OLTP engine to the user table partitions on each AP storage node. As an example, updates can be modeled as deletions following inserts. Therefore, in the logical log, there may only be inserts and deletions, without updates. Logical logs arriving at each AP storage node (such as logical log 500) can be categorized, persisted, and then applied sequentially to the in-memory incremental repository for each partition.

[0066] Figure 6An exemplary data structure 600 for the incremental repository is illustrated. As discussed above, each table in each partition has an in-memory incremental repository. Each data structure 600 contains two separate lists of variables classified by LSNs: an insertion variable list 602 and a deletion variable list 604. To speed up the search of the deletion variable list 604 (both the scan of the insertion variable list 602 and the base repository requires checking whether a row has been deleted in the deletion variable list 604), a hash map 606 (primary key, LSN list) is used on the deletion variable list 604 to quickly find deletions (O(1) cost for deletion checks).

[0067] Update operations can be replaced by delete operations, followed by insert operations, which delete rows with old data and insert rows with new data. This simplifies the HTAP storage engine, allowing it to handle only inserts and deletes. Incremental repository data is generated solely by log distributors. Each log distributor applies logs in batches to the incremental repository for improved efficiency and data freshness. To process logical log entries (if it's an insert log), the log can be parsed into tuples, and these tuples can be appended to the end of the insert list. If it's a delete log, to process logical log entries, the log can be parsed into tuples, the primary key tuple can be appended to the end of the delete list, and then <primary key, LSN list> can be inserted into the delete map.

[0068] Because the incremental repository resides entirely in storage, a background flush task periodically flushes rows to disk to make storage available. This may occur when enough rows are available (e.g., the number of rows > 60K or the accumulated data size exceeds 32MB) and / or when a sufficiently long time has elapsed since the last flush (e.g., 1 hour or any other scheduled time period). To avoid blocking query execution, flushed rows may remain in the incremental repository for a period of time, and those flushed rows will be truncated once all existing reads (including queries and background flushing operations) have completed before the flush point.

[0069] Therefore, concurrent flushing of query scans, compaction, and garbage collection (GC) operations on the incremental repository is permissible. To handle all partitions in a storage node, we can use multiple threads to perform flushing in parallel on different partitions. Long-running queries may block flushing. If the incremental repository becomes too large, we can cancel such queries and reschedule them to slots with fewer DML operations. For concurrent execution of log application, flushing, GC, and scans on the incremental repository, a safe solution could be to apply read / write locks to the incremental repository's data structures. Lock-free data structures and atomic compare-and-swap (CAS) instructions can also improve concurrency.

[0070] To efficiently manage incremental repositories in memory, a vector of fields can be used for memory management of these repositories. Each incremental repository initially has a field with a configurable starting size (e.g., 4096 bytes). If an incremental repository runs out of memory in its current field, a new field of double the size can be allocated and added to the vector. Each field is associated with an LSN that is the last LSN saved in that field. During incremental repository GC, we free those fields associated with LSNs smaller than the GCLSN.

[0071] Incremental repositories are crucial for high data freshness in HTAP systems. Hash mapping 606 during deletion flushes row-formatted data to column-formatted data, and the concurrency of log applications, flushes, and scans facilitate efficient analytical processing. Using sites for storage management helps improve both data freshness and efficient analytical processing.

[0072] Figure 7 The diagram illustrates an exemplary data layout 700 for table 702 in the base repository. When data is flushed from the incremental repository to the base repository, it is stored in columnar format and partitioned into multiple partitions 704a-c based on the partition key for efficient querying of OLAP workloads. Each HTAP storage node can contain multiple partitions, but each partition can only belong to one storage node. The base repository data resides in the same location as its associated incremental repository for efficient flushing and scanning operations. Data is stored in PAX format on the local file system of each storage node. Each table is partitioned by its partition key, and each partition can have multiple data blocks 706a-c. Within each data block, each column 710a-b, 714a-b is stored in a file, and metadata files 708, 712 contain metadata for all columns of this data block. The metadata may include the number of rows in the data block, the minimum / maximum / sum values ​​for each column, and a null map, etc. Data within each data block is sorted by the primary key.

[0073] A key-value (KV) system can be used to store deleted bitmaps containing information about rows removed from the base repository after a flush from the incremental repository. This allows columnstore data files to be immutable once written to disk. KV systems provide a fast, reliable, and simple way to insert and look up deletions on a specific key or a series of keys without using excessive memory space. In one embodiment, a single KV entry can be used to store a bitmap for each data block. In other embodiments, more than one KV entry can be used to store the bitmap for each data block. For each entry, we use the block ID of the data block (more specifically, the logical ID of the encoded primary key column file) as the key and the bitmap (bytes) as the value.

[0074] Example operations include, but are not limited to, insert / delete and scan with deletion. Insert / delete operations search for blocks with a deletion key, using the block ID to retrieve the bitmap. The sequence offset of the deletion key can be searched within the block, and the bitmap can be updated accordingly. Scan with deletion operations perform a normal scan on each block based on the range of input keys and the offset range of the resulting data within the block. Bitmap segments can be extracted from the entire deletion bitmap and applied to the results. To support efficient scanning and data compression to reduce storage usage, data in each column can be divided into data blocks 706a-c (compression units). Data within a data block can be compressed and stored in its column file. For efficient searching of primary keys, a Bloom filter for the primary key can be used. If the primary key comprises multiple columns, additional column files may be available to store composite primary keys to improve scan efficiency.

[0075] A columnar format for the basic repository, featuring efficient data encoding and compression, facilitates efficient analytical processing. The removal of bitmaps is also crucial for both high data freshness and efficient analytical processing.

[0076] In this embodiment, to improve scan performance and reduce storage usage, a compaction operation can be applied to the base repository in a background task. Since data blocks within a partition may have overlapping primary key ranges, queries may need to examine many blocks, which can significantly slow down query processing. Furthermore, rows marked as deleted in the deletion bitmap in the data files may be permanently deleted. This not only reduces the size of the base repository but also improves query processing efficiency. To achieve these two goals, the base repository performs a background compaction process to merge blocks within the partition. The compaction operation on the base repository improves scan performance and reduces storage usage. The compaction operation does not prevent other operations such as log application, flushing, GC, and scanning.

[0077] To enable the scanning and compaction processes to operate in parallel, at the start of a scan (each scan is marked with a read_LSN), the base repository can generate / get a snapshot of all visible data blocks it reads; this snapshot can be a list of block IDs for those blocks. For the compaction operation, older blocks can be made invisible at the compaction completion point and can also be marked with an allowed_truncated_LSN, which is the maximum read_LSN (maximum_read_LSN) of all active scans at the compaction completion point. After all active scans have completed with read_LSN <= allowed_truncated_LSN, those older blocks can be removed from the GC thread.

[0078] The basic algorithm used for the compaction process can include the following steps. At the start of compaction, data blocks can be selected for compaction based on strategies such as bitmap removal, key overlap, block size, IO budget, etc. Selected data blocks can be merged, and new data blocks (one or more) can be generated. Background merging does not prevent concurrent scanning.

[0079] Next, metadata may need to be updated to make the newly merged blocks visible (e.g., for OLAP engines). This time can be referred to as the compaction completion point. Metadata locking may be required for this operation. Scans that begin after the compaction completion point may be able to see the new blocks from the updated metadata file. Since active scans need to begin before the compaction completion point can complete scanning of the old data blocks, it prevents the deletion of old data blocks before all these scans are completed. Therefore, old data blocks can be organized / stored in a deletion set with tags (versions) that are the current maximum read_LSN at the compaction completion point. If one or more tags associated with an old data block are less than the current minimum read_LSN of all active scans, the background GC can delete the old data block from the deletion set.

[0080] To scan data within the storage engines, HTAP systems (which have one or more separate compute and storage engines) utilize scan operations. These scan operations are optimized for our HTAP storage engines through various techniques, such as asynchronous RPC calls, parallel scanning using a scan thread pool, predicate pushdown, aggregation pushdown, cost-based deletion filtering optimizations, continuous scanning of large data volumes, and partition pruning. These techniques, individually or in combination, contribute to efficient analytical processing.

[0081] Scan operations scan data in the storage engine via Remote Procedure Calls (RPCs) from the compute engine. Multiple scan operations can be executed in parallel by threads in a scan thread pool. Table scans can be split and distributed across multiple parallel scans of partitions of a table in the HTAP storage engine. Execution of scan operations within a partition can be asynchronous, and a callback function can be invoked to return the results when the scan is complete. The following might be the API for a scan operation: HTAP_scan(scan_id, table_id, partition_id, predicate list, projection list, read_LSN, result, callback_function);

[0082] When scanning data blocks in HTAP storage, predicate filters and projections of desired columns can be applied. If a query has aggregation operators, its partial aggregations can be pushed down to the scan of HTAP storage. Local partial aggregations within the storage node can significantly reduce the amount of data resulting from the scan. The design of predicate and aggregation pushdown is crucial for the performance of HTAP systems with separate compute and storage engines.

[0083] Additionally, it may be necessary to examine the deletion bitmap to exclude deleted rows. The results of the basic repository scan may also require excluding deleted rows in the incremental repository by looking up the deletion hash based on the row's primary key. Several suitable strategies exist for filtering out deleted data. One such strategy is to search the deletion hash for each row scanned. Another is to combine the deletion bitmap and deletions in the incremental repository for scanning the basic repository data. These two solutions have different costs for different workloads (e.g., the number of deletions in the incremental repository and the data size of the basic repository can affect the cost). Cost-based optimizations can be used to select when to filter out deletions in the incremental repository (if there are deletions in the incremental repository).

[0084] A single RPC call may not be enough to complete a scan. For example, each RPC might have row limitations, and if the scan volume is large, scan results must be returned in batches. To handle this continuous scan requirement, a continuous scan framework can be configured to handle scan operations with large data sizes in a compute and storage-separated HTAP system. The storage engine can save the scan state using iterator instances on snapshots of the incremental and base repositories and call the `nextBatch()` function to return the next batch of data. For example, the storage engine can maintain a mapping of `<scan ID, iterator instance>`. When the compute engine initiates a new scan request, the storage engine can assign a new scan ID, create an iterator instance, and return the scan ID as part of the response. For continuous scan requests, the storage engine can use the scan ID provided by the compute engine to locate the current iterator instance and continue from there.

[0085] Our OLAP engine's optimizer supports partition pruning. Based on this optimization, our HTAP storage engine may only scan partitions that haven't been pruned by the optimizer. This may not improve the performance of a single query; however, when multiple queries run concurrently, it can increase throughput and QPS (queries per second).

[0086] Figure 8 The diagram illustrates the storage engines that can be used in HTAP systems (e.g., Figure 4 The example process 800 executed by the storage subsystem 306 shown in the diagram. The storage engine can execute process 800 to receive and / or organize user data in a manner that can be utilized by the OLTP subsystem and the OLAP subsystem. Although in Figure 8 The operations are described as a sequence of operations, but those skilled in the art will understand that various embodiments may add, delete, reorder or modify the operations described.

[0087] At 802, data captured by a first computing system (such as in real-time or near real-time) can be received. The first computing system can be, for example, an OLTP engine (e.g., OLTP engine 303). Since OLTP workloads inherently perform better with row-based data formats, while OLAP workloads prefer columnar data formats, using a single format for both workloads means that the performance of the HTAP system will be affected.

[0088] To prevent this performance loss, user data may be stored in different formats for OLTP and OLAP engines. For example, user data may be stored in a first format (e.g., row format) for an OLTP engine and in a second format (e.g., column format) for an OLAP engine to enable efficient query processing. The storage subsystem may include multiple partitions. Each partition may include an incremental repository and a corresponding basic repository. For example, each partition may be further divided into an in-memory incremental repository and an on-disk basic repository.

[0089] At 804, multiple replicas can be distributed across the incremental repository by applying an arbitration protocol to multiple replicas of the logical log generated based on the data. The incremental repository can store data in a row-based format ordered by log sequence number (LSN, a sequence number / version indicating the order of data operations). Each user table can be partitioned based on a partitioning scheme defined at table creation time. Each partition can reside on a physical storage node (e.g., a server) and several copies of the data can be maintained for high availability. The logical log (such as the MySQL binary log) for committed DML transactions can be continuously dispatched from the OLTP engine to the user table partitions on each AP storage node. As an example, an update can be modeled as a deletion following an insert. Therefore, in the logical log, there may only be inserts and deletions, without updates.

[0090] Logical logs arriving at each AP storage node can be categorized, persistently stored, and then sequentially applied to an in-memory incremental repository for each partition. Data in the incremental repository is visible to queries performed by the online analytical processing (OLAP) engine. Multiple copies of the logical logs can be distributed, for example, by a log distributor (e.g., log distributor 404).

[0091] Data within the incremental repository can be flushed to the corresponding base repository. At point 806, data can be flushed from the incremental repository to the corresponding base repository based on one or more predefined rules. For example, data can be flushed from the incremental repository to the base repository when the data size (e.g., number of rows) exceeds a predefined threshold and / or after a fixed time interval. Data in the base repository can be stored in columnar format and is accessible by the OLAP engine. Data in the base repository can be divided into multiple partitions based on a partition key. Each partition in the base repository can include multiple data blocks, and the data in each of the multiple data blocks can be compressed. After flushing, the memory occupied by the flushed data can be collected by the trash can.

[0092] Data in the base repository may not be up-to-date in-place, so a deletion bitmap marking deleted rows can be maintained. As more and more data units are deleted and overlapped during flushing, compressing and rearranging the internal data can benefit storage and potential query performance; therefore, AP data units can be compressed periodically and redundant data cleaned up. Files in the base repository can be versioned, which is the largest LSN in the file at the time of flushing. The LSN may not be stored in the base repository, which could result in all files in the base repository having the same version (i.e., the last flushed LSN). This can offer several advantages. For example, it can save storage space, make deletion more efficient by using a deletion bitmap, and make scans faster when comparisons with LSNs are not required.

[0093] Figure 9 The diagram illustrates the storage engines that can be used in HTAP systems (e.g., Figure 4 The example process 900 is executed by the storage subsystem 306 shown in the diagram. The storage engine of the HTA system can execute process 900 to receive and / or organize user data in a manner that can be utilized by the OLTP and OLAP subsystems. Although in Figure 9 The operations are depicted as a sequence of operations, but those skilled in the art will understand that the various embodiments may add, remove, reorder or modify the depicted operations.

[0094] At position 902, data captured by a first computing system (such as in real-time or near real-time) can be received. The first computing system can be, for example, an OLTP engine (e.g., OLTP engine 303). Since OLTP workloads inherently perform better with row-based data formats, while OLAP workloads prefer columnar data formats, using a single format for both workloads means that the performance of the HTAP system will be affected.

[0095] To prevent this performance penalty, user data may be stored in different formats for OLTP and OLAP engines. For example, user data may be stored in a first format (e.g., row format) for OLTP engines and in a second format (e.g., column format) for OLAP engines to enable efficient query processing. The storage subsystem may include multiple partitions. Each partition may include an incremental repository and a corresponding basic repository. For example, each partition may be further divided into an in-memory incremental repository and an on-disk basic repository.

[0096] To prevent this performance penalty, user data may be stored in different formats for OLTP and OLAP engines. For example, user data may be stored in a first format (e.g., row format) for OLTP engines and in a second format (e.g., column format) for OLAP engines to enable efficient query processing. The storage subsystem may include multiple partitions. Each partition may include an incremental repository and a corresponding basic repository. For example, each partition may be further divided into an in-memory incremental repository and an on-disk basic repository.

[0097] At position 904, multiple replicas of the logical log generated based on the data can be distributed to the incremental repository by applying an arbitration protocol across multiple replicas. The incremental repository can store data in a row-based format ordered by log sequence number (LSN, a sequence number / version indicating the order of data operations). Each user table can be partitioned based on a partitioning scheme defined at table creation time. Each partition can reside on a physical storage node (e.g., a server), and several copies of the data can be maintained for high availability. The logical log (such as a MySQL binary log) for committed DML transactions can be continuously dispatched from the OLTP engine to the user table partitions on each AP storage node. Data in the incremental repository can be visible to queries performed by Online Analytical Processing (OLAP) engines. Multiple replicas of the logical log can be distributed, for example, by a log distributor (e.g., log distributor 404).

[0098] At position 906, data in the incremental repository can be updated by performing both delete and insert operations. For example, an update operation can be modeled as a delete operation following an insert operation. Therefore, in the logical log, there might only be inserts and deletes, without any updates.

[0099] To efficiently manage incremental repositories in memory, site vectors can be utilized. At 908, site vectors can be used to manage incremental repositories. A vector can include an initial site with a configurable starting size, and when an incremental repository runs out of memory in the initial site, a new site of double the size can be added to the vector. For example, each incremental repository initially has a site with a configurable starting size (e.g., 4096 bytes). If an incremental repository runs out of memory in the current site, a new site of double the size can be allocated and added to the vector. Each site is associated with an LSN that is the last LSN saved in that site. During incremental repository GC, we free those sites associated with LSNs smaller than the GCLSN.

[0100] Figure 10 The diagram illustrates the storage engines that can be used in HTAP systems (e.g., Figure 4 The example process 1000 executed by the storage subsystem 306 shown in the diagram. The storage engine of the HTAP system can execute process 1000 to receive and / or organize user data in a manner that can be utilized by the OLTP subsystem and the OLAP subsystem. Although in Figure 10 The operations are depicted as a sequence of operations, but those skilled in the art will understand that the various embodiments may add, remove, reorder or modify the depicted operations.

[0101] As discussed above, logical logs can reach each AP storage node and can be categorized, persistently stored, and then applied sequentially to the in-memory incremental repository for each partition. Once the logical logs arrive, they can be applied sequentially to the in-memory incremental repository. The latency between the time of data changes within the OLTP engine and the time when these changes occur within the incremental repository is virtually nonexistent (e.g., less than a few hundred milliseconds in most cases). The incremental repository can store data in a row-based format, sorted by log sequence number (LSN, a sequence number / version indicating the order of data operations).

[0102] At point 1002, data can be flushed from the incremental repository to the basic repository based on one or more predefined rules. Data within the incremental repository can be flushed to the basic repository periodically, for example, when its size exceeds a predefined threshold or after a fixed time interval. After flushing, the storage occupied by the flushed data can be collected by the trash. Data in the basic repository can be stored in columnar format and accessed by the OLAP engine. Data in the basic repository can be organized in columnar format for better performance of analytical workloads and persistently saved / stored on the local file system. Data in the basic repository can be divided into multiple partitions based on partition keys. Each partition in the basic repository can include multiple data blocks.

[0103] Data in the base repository may not be updatable in-place, so a deletion bitmap marking deleted rows can be maintained. At 1004, the deletion bitmap can be applied to at least one batch of data flushed from the incremental repository to the base repository. Because data blocks are not modified once generated, the KV system can be used to store the deletion bitmap, which contains information about the rows removed from the base repository after flushing from the incremental repository.

[0104] Each partition can be stored in a separate directory, and data for different columns can be stored in different files. Within each data file, data is categorized by primary key (one or more columns). Furthermore, data blocks can be periodically merged with a compaction process in a background task. At 1006, a compaction operation can be applied to the base repository to merge data blocks from one partition across multiple partitions. Because data blocks within a partition may have overlapping primary key ranges, queries may need to examine many blocks, significantly slowing down query processing. Additionally, rows marked as deleted in the deletion bitmap in the data file may be permanently deleted. This not only reduces the size of the base repository but also improves query processing efficiency. To achieve these two goals, the base repository performs a background compaction process to merge blocks within partitions. Compaction operations on the base repository improve scan performance and reduce storage usage. Compaction operations do not prevent other operations such as log application, flushing, GC, and scanning.

[0105] To enable the scanning and compaction processes to operate in parallel, at the start of a scan (each scan is marked with a read_LSN), the base repository can generate / get a snapshot of all visible data blocks it reads; this snapshot can be a list of block IDs for those blocks. For the compaction operation, older blocks can be made invisible at the compaction completion point and can also be marked with an allowed_truncated_LSN, which is the maximum read_LSN (maximum_read_LSN) of all active scans at the compaction completion point. After all active scans have completed with read_LSN <= allowed_truncated_LSN, those older blocks can be removed from the GC thread.

[0106] The basic algorithm used for the compaction process can include the following steps. At the start of compaction, data blocks can be selected for compaction based on strategies such as bitmap removal, key overlap, block size, IO budget, etc. Selected data blocks can be merged, and new data blocks (one or more) can be generated. Background merging does not prevent concurrent scanning.

[0107] Next, metadata may need to be updated to make the newly merged blocks visible (e.g., for OLAP engines). This time can be referred to as the compaction completion point. Metadata locking may be required for this operation. Scans that begin after the compaction completion point may be able to see the new blocks from the updated metadata file. Since active scans need to begin before the compaction completion point can complete scanning of the old data blocks, it prevents the deletion of old data blocks before all these scans are completed. Therefore, old data blocks can be organized / stored in a deletion set with tags (versions) that are the current maximum read_LSN at the compaction completion point. If one or more tags associated with an old data block are less than the current minimum read_LSN of all active scans, the background GC can delete the old data block from the deletion set.

[0108] To scan data within the storage engines, HTAP systems (which have one or more separate compute and storage engines) utilize scan operations. These scan operations are optimized for our HTAP storage engines in several ways, such as asynchronous RPC calls, parallel scanning using a scan thread pool, predicate pushdown, aggregation pushdown, cost-based deletion filtering optimizations, continuous scanning for large data volumes, and partition pruning. These techniques, individually or in combination, contribute to efficient analytical processing.

[0109] Scan operations scan data in the storage engine via Remote Procedure Calls (RPCs) from the compute engine. Multiple scan operations can be executed in parallel by threads in a scan thread pool. Table scans can be partitioned and distributed across multiple parallel scans of partitions of a table in the HTAP storage engine. Execution of scan operations within a partition can be asynchronous, and callback functions can be invoked to return results when the scan is complete. The following might be the API for a scan operation: HTAP_scan(scan_id, table_id, partition_id, predicate list, projection list, read)

[0110]

[0111] When scanning data blocks in HTAP storage, predicate filters and projections of desired columns can be applied. If a query has aggregation operators, a portion of its aggregation can be pushed down to the scan of HTAP storage. At 1008, in response to determining that an OLAP query includes aggregation operators indicating aggregation, at least a portion of the aggregation can be pushed down to scans of both the incremental and base repositories. Local partial aggregations within storage nodes can significantly reduce the amount of data resulting from the scan. The design of predicate and aggregation pushdown is critical to the performance of HTAP systems with separate compute and storage engines. At 1010, data blocks can be scanned by applying predicate filters and projection operations.

[0112] Additionally, it may be necessary to examine the deletion bitmap to exclude deleted rows. The results of the basic repository scan may also require excluding deleted rows in the incremental repository by looking up the deletion hash based on the row's primary key. Several suitable strategies exist for filtering out deleted data. One such strategy is to search the deletion hash for each row scanned. Another is to combine the deletion bitmap and deletions in the incremental repository for scanning the basic repository data. These two solutions have different costs for different workloads (e.g., the number of deletions in the incremental repository and the data size of the basic repository can affect the cost). Cost-based optimizations can be used to select when to filter out deletions in the incremental repository (if there are deletions in the incremental repository).

[0113] A single RPC call may not be able to complete a scan. For example, each RPC may have row limitations, and if the scan volume is large, the scan results must be returned in batches. To handle this continuous scan requirement, a continuous scan framework can be configured to handle scan operations with large data sizes in a compute- and storage-separated HTAP system.

[0114] At position 1012, continuous scans can be performed on the incremental and base repositories based on the mapping of scan IDs and iterator instances. The storage engine can save the scan state using iterator instances on snapshots of the incremental and base repositories and call the `nextBatch()` function to return the next batch of data. For example, the storage engine can save a mapping of `<scan ID, iterator instance>`. When the compute engine initiates a new scan request, the storage engine can assign a new scan ID, create an iterator instance, and return the scan ID as part of the response. For continuous scan requests, the storage engine can use the scan ID provided by the compute engine to locate the current iterator instance and continue from there.

[0115] Our OLAP engine optimizer supports partition pruning. Based on this optimization, our HTAP storage engine may only scan partitions that were not pruned by the optimizer. This may not improve the performance of a single query; however, when multiple queries run concurrently, it can increase throughput and QPS (queries per second). At 1014, scan results can be returned in batches or multiple batches.

[0116] Figure 11 The illustration shows computing devices that can be used in various aspects, such as Figure 1 The services, networks, modules, and / or devices described herein. About Figure 1 In the example architecture, cloud network 102, network 120, client devices 104a-d, service 102, HTAP system 100, and / or node 108 can all be... Figure 11 This is implemented using one or more instances of the computing device 1100. Figure 11 The computer architecture shown illustrates conventional server computers, workstations, desktop computers, laptop computers, tablet computers, network devices, PDAs, e-readers, digital cellular phones, or other computing nodes, and can be used to perform any aspect of the computer described herein, such as implementing the methods described herein.

[0117] The computing device 1100 may include a substrate or “motherboard” as a printed circuit board, to which multiple components or devices may be connected via a system bus or other electrical communication path. One or more central processing units (CPUs) 1104 may operate in conjunction with a chipset 1106. The CPUs 1104 may be standard programmable processors that perform the arithmetic and logic operations required to perform the operations of the computing device 1100.

[0118] One or more CPU 1104s can perform necessary operations by manipulating switching elements that distinguish and change these states, transitioning from one discrete physical state to the next. A switching element typically includes electronic circuitry (such as a flip-flop) that holds one of two binary states, and electronic circuitry that provides the output state based on a logical combination of the states of one or more other switching elements (such as logic gates). These basic switching elements can be combined to create more complex logic circuits, including registers, adder-subtractor units, arithmetic logic units, floating-point units, etc.

[0119] One or more CPUs 1104 can be expanded or replaced by other processing units, such as one or more GPUs 1105. One or more GPUs 1105 may include processing units dedicated to, but not limited to, highly parallel computing, such as graphics and other visualization-related processing.

[0120] Chipset 1106 may provide an interface between CPU(s) 1104 and the remaining components and devices on the substrate. Chipset 1106 may provide an interface to random access memory (RAM) 1108, which serves as main memory in computing device 1100. Chipset 1106 may further provide an interface to a computer-readable storage medium, such as read-only memory (ROM) 1120 or non-volatile RAM (NVRAM) (not shown), for storing basic routines that can help start computing device 1100 and transfer information between various components and devices. ROM 1120 or NVRAM may also store other software components necessary for the operation of computing device 1100 according to the aspects described herein.

[0121] Computing device 1100 can operate in a networked environment using a logical connection via a local area network (LAN) to remote computing nodes and computer systems. Chipset 1106 may include functionality for providing network connectivity via a network interface controller (NIC) 1122 (such as a Gigabit Ethernet adapter). NIC 1122 enables computing device 1100 to connect to other computing nodes via network 1116. It should be understood that multiple NICs 1122 may be present in computing device 1100 to connect the computing device to other types of networks and remote computer systems.

[0122] Computing device 1100 can be connected to mass storage device 1128, which provides non-volatile storage for the computer. Mass storage device 1128 can store system programs, application programs, other program modules, and data, which have been described in more detail herein. Mass storage device 1128 can be connected to computing device 1100 via storage controller 1124, which is connected to chipset 1106. Mass storage device 1128 can consist of one or more physical storage units. Mass storage device 1128 may include management unit 1111. Storage controller 1124 can interface with physical storage units via a Serial Attached SCSI (SAS) interface, a Serial Advanced Technology Attachment (SATA) interface, a Fibre Channel (FC) interface, or other types of interfaces used for physical connection and data transfer between the computer and physical storage units.

[0123] The computing device 1100 can store data on the mass storage device 1128 by transforming the physical state of the physical storage units to reflect the stored information. The specific transformation of the physical state can depend on various factors and different implementations described herein. Examples of such factors may include, but are not limited to, the technology used to implement the physical storage units and whether the mass storage device 1128 is characterized as main memory or auxiliary memory.

[0124] For example, computing device 1100 can store information in mass storage device 1128 by issuing instructions via storage controller 1124 to change the magnetic properties of a specific location in a disk drive unit, the reflection or refraction properties of a specific location in an optical storage unit, or the electrical properties of a specific capacitor, transistor, or other discrete component in a solid-state storage unit. Other transformations of the physical medium are possible without departing from the scope and spirit of this description; the foregoing examples are provided merely for the purpose of illustration. Computing device 1100 can further read information from mass storage device 1128 by detecting the physical state or characteristics of one or more specific locations within the physical storage unit.

[0125] In addition to the high-capacity storage device 1128 described above, the computing device 1100 can also access other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. Those skilled in the art will understand that the computer-readable storage medium can be any available medium that provides storage for non-transitory data and can be accessed by the computing device 1100.

[0126] By way of example and not limitation, computer-readable storage media may include volatile and non-volatile, transient and non-transient computer-readable storage media implemented in any method or technology, as well as removable and non-removable media. Computer-readable storage media includes, but is not limited to, RAM, ROM, erasable programmable ROM (“EPROM”), electrically erasable programmable ROM (“EEPROM”), flash memory or other solid-state storage technologies, optical disc ROM (“CD-ROM”), digital versatile disc (“DVD”), high-definition DVD (“HD-DVD”), BLU-RAY or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, other magnetic storage devices, or any other media that can be used to store desired information in a non-transient manner.

[0127] High-capacity storage devices (such as) Figure 11 The mass storage device 1128 depicted herein can store an operating system used to control the operation of the computing device 1100. The operating system may include a version of the Linux operating system. The operating system may include a version of the Windows Server operating system from Microsoft Corporation. According to a further aspect, the operating system may include a version of the UNIX operating system. Various mobile phone operating systems, such as iOS and Android, may also be utilized. It should be understood that other operating systems may also be utilized. The mass storage device 1128 can store other systems, applications, and data utilized by the computing device 1100.

[0128] Mass storage device 1128 or other computer-readable storage media may also be encoded with computer-executable instructions that, when loaded into computing device 1100, transform the computing device from a general-purpose computing system into a special-purpose computer capable of implementing the aspects described herein. As described above, these computer-executable instructions transform computing device 1100 by specifying how CPU(s)(one or more) 1104 transition between states. Computing device 1100 can access the computer-readable storage medium storing the computer-executable instructions, which, when executed by computing device 1100, can perform the methods described herein.

[0129] Computing devices (such as Figure 11The computing device 1100 depicted may also include an input / output controller 1132 for receiving and processing input from multiple input devices, such as a keyboard, mouse, touchpad, touchscreen, electronic pen, or other types of input devices. Similarly, the input / output controller 1132 may provide output to a display, such as a computer monitor, flat panel display, digital projector, printer, plotter, or other types of output device. It should be understood that the computing device 1100 may not include... Figure 11 All components shown may include Figure 11 Other components not explicitly shown in the document, or those that can be utilized with Figure 11 The architecture shown is completely different from the one described above.

[0130] As described herein, a computing device can be a physical computing device, such as... Figure 11 The computing device 1100. A computing node may also contain virtual machine host processes and one or more virtual machine instances. Computer-executable instructions can be indirectly executed by the physical hardware of the computing device by interpreting and / or executing instructions stored and executed in the context of the virtual machine.

[0131] It should be understood that these methods and systems are not limited to specific methods, specific components, or specific implementations. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

[0132] As used in the specification and appended claims, the singular forms “a,” “an,” and “the” include plural indicators unless the context explicitly specifies otherwise. A range herein may be expressed as from “about” one particular value and / or to “about” another particular value. When expressing such a range, another embodiment includes from one particular value and / or to another particular value. Similarly, when a value is expressed as an approximation using the antecedent “about,” it should be understood that the particular value forms another embodiment. It should further be understood that each endpoint in the range is significant both relative to and independent of the other endpoint.

[0133] "Optional" or "optionally" means that the event or situation described below may or may not occur, and the description includes instances where the event or situation occurs and instances where the event or situation does not occur.

[0134] Throughout the description and claims of this specification, the word “comprising” and variations thereof, such as “comprising” and “comprises”, mean “including but not limited to” and are not intended to exclude, for example, other components, integrals, or steps. “Exemplary” means “example” and is not intended to convey indications of preferred or ideal embodiments. “Like” is not used in a limiting sense but for interpretive purposes.

[0135] Components that can be used to perform the described methods and systems are described. When describing combinations, subsets, interactions, groups, etc., of these components, it should be understood that, although specific references to each of the various individual and collective combinations and arrangements of these components may not be explicitly described, each is specifically considered and described herein for all methods and systems. This applies to all aspects of this application, including but not limited to the operations in the described methods. Therefore, if multiple additional operations exist that can be performed, it should be understood that each of these additional operations can be performed using any particular embodiment or combination of embodiments of the described methods.

[0136] The method and system can be more easily understood by referring to the following detailed implementation of preferred embodiments and the examples contained therein, as well as the accompanying drawings and descriptions.

[0137] As those skilled in the art will understand, the methods and systems may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, these methods and systems may take the form of a computer program product on a computer-readable storage medium having computer-readable program instructions (e.g., computer software) implemented in the storage medium. More particularly, the methods and systems may take the form of network-implemented computer software. Any suitable computer-readable storage medium may be utilized, including hard disks, CD-ROMs, optical storage devices, or magnetic storage devices.

[0138] Embodiments of the methods and systems are described below with reference to block diagrams and flowcharts of methods, systems, apparatuses, and computer program products. It should be understood that each block in the block diagrams and flowcharts, as well as combinations of blocks in the block diagrams and flowcharts, can be implemented accordingly by computer program instructions. These computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart.

[0139] These computer program instructions may also be stored in a computer-readable storage medium, which may direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of writing containing computer-readable instructions for implementing the functions specified in one or more boxes of the flowchart. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus, resulting in a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more boxes of the flowchart.

[0140] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Furthermore, some method or process blocks may be omitted in some implementations. The methods and processes described herein are not limited to any particular order, and the associated blocks or states may be executed in other suitable orders. For example, the described blocks or states may be executed in an order different from that specifically described, or multiple blocks or states may be combined in a single block or state. Example blocks or states may be executed serially, in parallel, or in some other manner. Blocks or states may be added to or removed from the described example embodiments. The example systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the described example embodiments.

[0141] It should also be understood that various items are shown to be stored in memory or on storage devices when in use, and these items, or portions thereof, may be transferred between memory and other storage devices for memory management and data integrity purposes. Alternatively, in other embodiments, some or all of the software modules and / or systems may be executed in memory on another device and communicate with the illustrated computing system via inter-computer communication. Furthermore, in some embodiments, some or all of the systems and / or modules may be implemented or provided in other ways (such as at least partially in firmware and / or hardware), including, but not limited to, one or more application-specific integrated circuits (“ASICs”), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field-programmable gate arrays (“FPGAs”), complex programmable logic devices (“CPLDs”), etc. Some or all of the modules, systems, and data structures may also be stored (e.g., as software instructions or structured data) on computer-readable media (such as hard disks, memory, networks, or portable media products) for retrieval by appropriate devices or via appropriate connections. Systems, modules, and data structures can also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) over a variety of computer-readable transmission media, including wireless and wired / cable-based media, and can take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, such computer program products can also take other forms. Therefore, the invention can be practiced with other computer system configurations.

[0142] Although methods and systems have been described in conjunction with preferred embodiments and specific examples, this is not intended to limit the scope to the specific embodiments illustrated, as the embodiments herein are intended in all respects to be illustrative rather than restrictive.

[0143] Unless otherwise expressly stated, it is not intended that any method described herein be construed as requiring its operations to be performed in a particular order. Therefore, where a method claim does not actually describe the order in which its operations are followed, or where the claims or specification do not otherwise specifically state that the operations will be limited to a particular order, it is not intended that the order be inferred from any particular perspective. This applies to any possible non-express basis of interpretation, including: logical questions concerning the arrangement of steps or the flow of operations; simple meanings derived from grammatical organization or punctuation; and the number or type of embodiments described in the specification.

[0144] It will be apparent to those skilled in the art that various modifications and variations can be made without departing from the scope or spirit of this disclosure. Other embodiments will be apparent to those skilled in the art upon consideration of the specification and practice described herein. The specification and example figures are to be considered exemplary only, wherein the true scope and spirit are indicated by the appended claims.

Claims

1. A method for storage, comprising: Receive data captured by a first processing engine, wherein the first processing engine is configured to perform online transaction processing; By applying an arbitration protocol to multiple copies of a logical log generated based on the data, the multiple copies are distributed to an incremental repository in the storage system, wherein the data in the incremental repository is stored in row format and is visible to queries performed by an online analytics processing engine. as well as Based on one or more predetermined rules, data is flushed from the incremental repository to the basic repository in the storage system, wherein the data in the basic repository is stored in columnar format and can be accessed by the second processing engine, and the data in the basic repository is divided into multiple partitions based on partition keys, and each partition in the basic repository includes multiple data blocks. The storage system is configured to persistently store the same data in different formats for use by the first processing engine and the second processing engine, respectively. The storage system, the first processing engine, and the second processing engine are configured to be decoupled from each other.

2. The method according to claim 1, further comprising: The data in the incremental repository is updated by performing delete and insert operations.

3. The method of claim 1, wherein the incremental repository includes an insertion variable list and a deletion variable list, and the insertion variable list and the deletion variable list are categorized based on log sequence numbers.

4. The method of claim 1, wherein each of the plurality of data blocks comprises a plurality of column files and a metadata file, and the metadata file comprises metadata associated with the plurality of column files.

5. The method according to claim 1, further comprising: A deletion bitmap is applied to at least one batch of data flushed from the incremental repository to the base repository, wherein the deletion bitmap includes information indicating the rows that are removed after flushing from the incremental repository.

6. The method according to claim 1, further comprising: A compaction operation is applied to the base repository to merge data blocks from one of the multiple partitions.

7. The method according to claim 1, further comprising: The multiple data blocks are scanned by applying predicate filters and projection operations.

8. The method according to claim 1, further comprising: In response to determining that the online analytical processing query includes an aggregation operator indicating aggregation, at least a portion of the aggregation is pushed down to the scan of the incremental repository and the base repository.

9. The method according to claim 1, further comprising: Based on the mapping between scan identification and iterator instances, a continuous scan is performed on the incremental repository and the basic repository; as well as Return scan results in batches.

10. A system for storage, comprising: At least one processor; as well as At least one memory, communicatively coupled to the at least one processor, and including instructions that, when executed by the at least one processor, cause the system to perform operations including: Receive data captured by a first processing engine, wherein the first processing engine is configured to perform online transaction processing; By applying an arbitration protocol to multiple copies of a logical log generated based on the data, the multiple copies are distributed to an incremental repository in the storage system, wherein the data in the incremental repository is stored in row format and is visible to queries performed by an online analytical processing engine; and Based on one or more predetermined rules, data is flushed from the incremental repository to the basic repository in the storage system, wherein the data in the basic repository is stored in columnar format and can be accessed by the second processing engine, and the data in the basic repository is divided into multiple partitions based on partition keys, and each partition in the basic repository includes multiple data blocks. The storage system is configured to persistently store the same data in different formats for use by the first processing engine and the second processing engine, respectively. The storage system, the first processing engine, and the second processing engine are configured to be decoupled from each other.

11. The system of claim 10, further comprising: The data in the incremental repository is updated by performing delete and insert operations.

12. The system of claim 10, wherein the incremental repository includes an insertion variable list and a deletion variable list, and the insertion variable list and the deletion variable list are classified based on log sequence numbers (LSNs).

13. The system of claim 10, wherein each of the plurality of data blocks comprises a plurality of column files and a metadata file, and the metadata file comprises metadata associated with the plurality of column files.

14. The system of claim 10, further comprising: A deletion bitmap is applied to at least one batch of data flushed from the incremental repository to the base repository, wherein the deletion bitmap includes information indicating the rows that are removed after flushing from the incremental repository.

15. The system according to claim 10, further comprising: A compaction operation is applied to the base repository to merge data blocks from one of the multiple partitions.

16. The system of claim 10, further comprising: The multiple data blocks are scanned by applying predicate filters and projection operations.

17. The system of claim 10, further comprising: In response to determining that the online analytical processing query includes an aggregation operator indicating aggregation, at least a portion of the aggregation is pushed down to the scan of the incremental repository and the base repository.

18. The system of claim 10, further comprising: Based on the mapping between scan identification and iterator instances, a continuous scan is performed on the incremental repository and the basic repository; as well as Return scan results in batches.

19. A non-transitory computer-readable storage medium comprising computer-readable instructions that, when executed by a system, cause the system to perform operations, the operations including: Receive data captured by a first processing engine, wherein the first processing engine is configured to perform online transaction processing; By applying an arbitration protocol to multiple copies of a logical log generated based on the data, the multiple copies are distributed to an incremental repository in the storage system, wherein the data in the incremental repository is stored in row format and is visible to queries performed by the second processing engine for online analytical processing; as well as Based on one or more predetermined rules, data is flushed from the incremental repository to the basic repository in the storage system, wherein the data in the basic repository is stored in columnar format and can be accessed by the second processing engine, and the data in the basic repository is divided into multiple partitions based on partition keys, and each partition in the basic repository includes multiple data blocks. The storage system is configured to persistently store the same data in different formats for use by the first processing engine and the second processing engine, respectively. The storage system, the first processing engine, and the second processing engine are configured to be decoupled from each other.

20. The non-transitory computer-readable storage medium according to claim 19, The incremental repository includes a list of inserted variables and a list of deleted variables, and the list of inserted variables and the list of deleted variables are categorized based on log sequence numbers; and Each of the plurality of data blocks includes a plurality of column files and a metadata file, and the metadata file includes metadata associated with the plurality of column files.

Images