Hybrid Data Processing Systems and Methods

Abstract

This disclosure describes a Hybrid Transactional and Analytical Processing (HTAP) technology. An HTAP system includes: a first processing engine configured to perform online transaction processing, a second processing engine configured to perform online analytical processing, and a storage device communicating with the first and second processing engines. The first processing engine, the second processing engine, and the storage device may be modularized and configured to be decoupled from each other. The system may be configured to have the first processing engine capture data in real time, organize data in a first component of the storage device according to a first format for use by the first processing engine, propagate data to a second component of the storage device system, and organize data in a second format in the second component of the storage device for use by the second processing engine.

Description

Background Technology

[0001] Data processing refers to the process of performing specific operations on a set of data or a database. A database is an organized collection of facts and information, such as records about inventory, customers, etc. Various forms of data processing exist and serve diverse applications in business settings. As databases are increasingly used to store large amounts of complex data, improvements in data processing technologies may be expected. Attached Figure Description

[0002] The following detailed description will be better understood when read in conjunction with the accompanying drawings. For illustrative purposes, exemplary embodiments of various aspects of this disclosure are shown in the drawings; however, the invention is not limited to the specific methods and means disclosed.

[0003] Figure 1 An example system including cloud services is shown.

[0004] Figure 2 An example system including more than one cloud service is shown.

[0005] Figure 3A shows an example system that combines transactional and analytical processing.

[0006] Figure 3B shows another example system that combines transactional and analytical processing.

[0007] Figure 4 An example system for online transaction processing is shown.

[0008] Figure 5 An example system for online analytical processing is shown.

[0009] Figure 6 An example system of data storage device is shown.

[0010] Figure 7 An example system that combines transactional and analytical processing is shown.

[0011] Figure 8 An example procedure for performing mixed transactional and analytical processing is shown.

[0012] Figure 9 An example process for performing a centralized metadata service is shown.

[0013] Figure 10 An example procedure for performing mixed transactional and analytical processing is shown.

[0014] Figure 11 An example computing device is shown that can be used to perform any of the techniques disclosed herein. Detailed Implementation

[0015] Online Analytical Processing (OLAP) systems enable users to interactively analyze multidimensional data from multiple perspectives. Multidimensional data includes datasets with three or more dimensions. OLAP systems allow users to analyze information from multiple database systems simultaneously. OLAP systems enable analysts to extract and view data, such as business data, from different angles. Analysts often need to group, aggregate, and combine data. These OLAP operations in data mining are resource-intensive. With OLAP, data can be pre-computed and pre-aggregated, making analysis faster. However, traditional OLAP systems typically load large amounts of data in batches periodically. This can make OLAP systems susceptible to outdated data.

[0016] OLAP typically contrasts with OLTP (Online Transaction Processing) systems. OLTP systems capture, store, and process data from transactions. A common characteristic of OLTP systems is the large volume of relatively uncomplex queries used to process transactions, rather than for business intelligence or reporting purposes. Traditional OLTP systems support Data Manipulation Language (DML), a computer programming language used to add (insert), delete, and modify (update) data in the database. Traditional OLTP systems also effectively support point lookup queries.

[0017] Significant differences exist between OLAP and OLTP systems. For example, OLTP systems typically lack massively parallel query engines (like those in OLAP systems) to support efficient and complex query processing of large amounts of data. As another example, OLAP systems are primarily optimized for read-only scenarios and may not support other types of queries, while OLTP systems handle all types of queries (read, insert, update, and delete). As yet another example, OLTP systems are associated with short atomic transactions, while OLAP systems allow for more flexible distribution patterns and higher scalability, but with increased latency and no guaranteed upper limit on processing time.

[0018] Many mainstream data platforms / systems focus on only one of these workloads (e.g., OLAP or OLTP). However, many scenarios (e.g., business scenarios) require both complex OLAP-style analysis of newly imported data and the transactional support and robust data consistency provided by OLTP systems. Hybrid Transactional / Analytical Processing (HTAP) systems are best suited for such scenarios.

[0019] HTAP systems offer several unique advantages. In an HTAP system, OLAP and OLTP workloads are integrated into a single system. By integrating OLAP and OLTP workloads into a single system, the complexity and cost of deployment and maintenance are significantly reduced. Such a general-purpose system significantly reduces staleness in query results (which is typically introduced by time-consuming and expensive ETL processes from the operational database to the data warehouse). These systems also have the ability to perform complex analyses of real-time data, thus addressing modern business models that often require efficient responses to fleeting opportunities that must be capitalized in real time.

[0020] However, many existing HTAP systems have drawbacks. First, many existing HTAP systems integrate OLAP query engines as extensions of transactional databases, which makes it difficult to prevent interference between analytical queries and transactional workloads. Users / designers may have to make trade-offs between read consistency and analytical performance in such systems. Second, many existing HTAP systems choose to maintain only one type of data, thus requiring a fixed format for the underlying data. Since OLTP workloads inherently perform better with row-based data formats, while OLAP workloads prefer column-based data formats, using the same format for both workloads means the HTAP system's performance will suffer. Third, many HTAP systems use in-memory data warehouses to handle certain (OLTP or OLAP) workloads. Due to the heavy use of memory, such systems are typically expensive. If data cannot be adapted to memory, the performance of such systems will be a real concern. Therefore, HTAP systems that address these drawbacks are needed.

[0021] This paper describes an HTAP system capable of handling business scenarios with both transactional (OLTP) and complex analytical (OLAP) workloads. Unlike existing HTAP systems, the HTAP system described in this paper utilizes a large-scale real-time analytics architecture that facilitates fresh data changes and strong data consistency. The improved HTAP system has the ability to coordinate large amounts of compute and storage resources (or even more horizontal scaling capabilities) to handle large-scale user workloads. It assumes large data volumes that are not suitable for in-memory storage and require data persistence. The improved HTAP system provides fast (e.g., real-time) Data Manipulation Language (DML) query processing on top of the row data warehouse in the OLTP engine. The improved HTAP system also facilitates distributed query processing for complex OLAP queries, including joins, aggregations, etc. The improved HTAP system also facilitates fresh data changes and strong data consistency. Data changes are continuously applied to the distributed incremental data warehouse in in-memory. This allows the OLAP engine to query the latest OLTP data changes. The improved HTAP system provides global snapshot isolation across OLTP and OLAP engines and supports OLAP queries to read the latest updates in the current system, thus ensuring strong data consistency.

[0022] The improved HTAP system features a flexible, modular design. The system's main components, such as the OLTP query engine, OLAP query engine, and / or underlying storage, are modularized and decoupled. Therefore, components can be easily changed without modifying the entire architecture. The modular design also makes horizontal scaling of each component easier.

[0023] HTAP systems, such as the improved HTAP systems described above, can be utilized by a variety of different systems or entities. Figure 1 The illustration shows an example system 100 including an HTAP system. System 100 may include a cloud network 102 and multiple client devices 104a-d. The cloud network 102 and the multiple client devices 104a-d may communicate with each other via one or more networks 120.

[0024] Cloud network 102 may be located in a data center, such as a single site, or distributed across different geographical locations (e.g., in several sites). Cloud network 102 may provide services via one or more networks 120. Network 120 includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or similar items. Network 120 may include physical links, such as coaxial cable links, twisted-pair cable links, fiber optic links, combinations thereof, and / or similar items. Network 120 may include wireless links, such as cellular links, satellite links, Wi-Fi links, and / or similar items.

[0025] Cloud network 102 may include multiple computing nodes 118 hosting various services. In an embodiment, node 118 hosts service 112. Service 112 may include content streaming services, such as Internet Protocol (IP) video streaming services. Service 112 may be configured to distribute content via various transport technologies. Service 112 is configured to provide content such as video, audio, text data, combinations thereof, and / or similar items. Content may include content streams (e.g., video streams, audio streams, information streams), content files (e.g., video files, audio files, text files), and / or other data. Content may be stored in a database. For example, service 112 may include video sharing services, video hosting platforms, content distribution platforms, collaborative gaming platforms, and / or similar items. In addition to, or in lieu of, content streaming services, service 112 may include any other type of service.

[0026] In this embodiment, service 112 may be provided to client device 104 via network 120. If service 112 is a content streaming service, content may be output to different client devices 104 via network 120. Content may be streamed to client devices 104. The content stream may be a stream of short videos received from service 112. Multiple client devices 104 may be configured to access content from service 112. In this embodiment, client device 104 may include an application. The application outputs (e.g., displays, renders, presents) content to a user associated with client device 104. The content may include video, audio, comments, text data, and / or similar items.

[0027] In this embodiment, a user can use an application on client device 104 to create content and upload short videos to cloud network 102. Client device 104 can access an interface of the application. This interface may include input elements. For example, the input elements may be configured to allow the user to create content. To create content, the user may grant the application permission to access an image capture device, such as the camera or microphone of client device 104. After the user has created content, the user can use the application to upload the content to cloud network 102 and / or save the content locally to user device 104. Service 112 may store the uploaded content and any metadata associated with the content in one or more databases.

[0028] Multiple client devices 104 may include any type of computing device, such as mobile devices, tablets, laptops, desktop computers, smart TVs or other smart devices (e.g., smartwatches, smart speakers, smart glasses, smart helmets), gaming devices, set-top boxes, digital streaming devices, robots, and / or similar items. Multiple client devices 104 may be associated with one or more users. A single user may use one or more of the multiple client devices 104 to access the cloud network 102. Multiple client devices 104 may travel to multiple locations and use different networks to access the cloud network 102.

[0029] Multiple compute nodes 118 can handle tasks associated with service 112. Multiple compute nodes 118 can be implemented as one or more compute devices, one or more processors, one or more virtual compute instances, combinations thereof, and / or similar items. Multiple compute nodes 118 can be implemented by one or more compute devices. One or more compute devices may include virtualized compute instances. Virtualized compute instances may include virtual machines, such as emulations of computer systems, operating systems, servers, and / or similar items. Virtual machines may be loaded by the compute device based on virtual images and / or other data defining specific software (e.g., operating systems, dedicated applications, servers) used for emulation. As the demand for different types of processing services changes, different virtual machines may be loaded and / or terminated on one or more compute devices. A hypervisor may be implemented to manage the use of different virtual machines on the same compute device.

[0030] In this embodiment, service 112 includes HTAP system 110. HTAP system 110 may include multiple different components (e.g., subsystems). For example, HTAP system 110 may include one or more of a transactional OLTP engine, an analytics OLAP engine, an underlying decoupled storage device, a metadata service, and / or an intelligent agent. The architecture of HTAP system 110 will be referenced below. Figures 3a-7 A more detailed discussion will follow. Additional details about each subsystem within the subsystem will also be referenced below. Figures 3a-7 To elaborate further.

[0031] HTAP system 110 may have an architecture that supports heterogeneous query engines. This architecture can handle both transactional OLTP workloads and complex analytical (OLAP) workloads. The architecture follows a modular design, and its main components are fully decoupled, providing flexibility and easy horizontal scaling. For example, components of HTAP system 110 can be easily changed into similar, established subsystems. This architecture eliminates interference between OLTP and OLAP workloads by having separate query processing engines and different data replicas.

[0032] HTAP system 110 can maintain user data in different formats for OLTP and OLAP engines. For example, for efficient query processing, HTAP system 110 can maintain user data in row format for the OLTP engine and in column format for the OLAP engine. This architecture can have a single point of truth for metadata and can use a separate metadata service to provide up-to-date metadata to certain components of HTAP system 110. The architecture of HTAP system 110 may include an intelligent agent that dispatches queries to OLTP and OLAP subsystems based on the nature of the query (and thus hides internal details from users / clients). Users / clients can utilize HTAP system 110 with a single unified interface. For example, users / clients may be able to utilize HTAP system 110 with an interface for client device 104. This architecture can support various APIs (e.g., ANSI SQL, JDBC, ODBC, etc.) based on user needs.

[0033] The architecture of HTAP system 110 is capable of handling large-scale data. This is a result of the fact that the compute and storage components in HTAP system 110 can be decoupled. A decoupled storage system capable of persisting large amounts of data is utilized because it is not assumed that the data will be able to fit in memory. The compute and storage resources in HTAP system 110 can also be horizontally scaled, and therefore can flexibly handle large amounts of data and large-scale (OLTP and OLAP) workloads.

[0034] The architecture of HTAP system 110 enables efficient real-time data processing. DML queries can be efficiently handled by the OLTP engine and written efficiently to the underlying storage in row-by-row format. The architecture of HTAP system 110 may include an OLAP query engine with distributed query processing capabilities (high parallelism, better resource utilization) to efficiently handle complex OLAP queries, including joins, aggregations, etc. The fact that interference between OLTP and OLAP workloads is minimized makes efficient and real-time processing possible because the architecture of HTAP system 110 facilitates the storage of distinct copies of data on the OLTP and OLAP sides. OLTP and OLAP data formats can be optimized separately to suit their workloads. A single source of data change can exist through HTAP system 110 (from the OLTP side), thereby simplifying the consistency model and concurrent processing across OLTP and OLAP components.

[0035] The architecture of HTAP System 110 provides fresh / real-time data changes for OLAP queries. Logical logs of DML operations are propagated from OLTP components to OLAP components immediately upon commit. These logs can be dispatched to distributed partitions and continuously applied to the in-memory incremental data warehouse (Delta Store) via typically very fast in-memory operations. Data changes carried by the logical logs are immediately available for OLAP queries when applied to the in-memory incremental data warehouse. The architecture of HTAP System 110 leverages unified versioning across HTAP System 110, ensuring strong data consistency. Like most transactional OLTP database engines, the OLTP components of HTAP System 110 support snapshot isolation and other (weaker) consistency models.

[0036] Although Figure 1 System 100 illustrates HTAP system 110 as being provided by a single cloud network 102, but the various components / subsystems of HTAP system 110 may be provided by multiple different cloud networks. Figure 2 The illustration shows an example system 200 including an HTAP system with components / subsystems spanning multiple cloud networks. System 200 may include cloud networks 202a-b and multiple client devices 204a-d. Cloud networks 202a-b and multiple client devices 204a-d may communicate with each other via one or more networks 220.

[0037] Each cloud network in Cloud Networks 202a-b can be similar to the one described above. Figure 1 The cloud network 102 described herein. Each cloud network in cloud networks 202a-b may be located in a data center, such as a single site, or distributed across different geographical locations (e.g., at several sites). Cloud networks 202a-b may provide services(s) via one or more networks 220. Cloud networks 202a-b include various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or similar items. Cloud networks 202a-b may include physical links, such as coaxial cable links, twisted-pair cable links, fiber optic links, combinations thereof, and / or similar items. Cloud networks 202a-b may include wireless links, such as cellular links, satellite links, Wi-Fi links, and / or similar items.

[0038] Each cloud network in cloud networks 202a-b may include multiple compute nodes hosting various services. In this embodiment, node hosting service 212a is associated with cloud network 202a, while node hosting service 212b is associated with cloud network 202b. Services 212a-b may include any type of service, such as those mentioned above. Figure 1 The content streaming service described.

[0039] Multiple client devices 204 may include any type of computing device, such as mobile devices, tablets, laptops, desktop computers, smart TVs or other smart devices (e.g., smartwatches, smart speakers, smart glasses, smart helmets), gaming devices, set-top boxes, digital streaming devices, robots, and / or similar items. Multiple client devices 104 may be associated with one or more users. A single user may use one or more of the multiple client devices 104 to access at least one of the cloud networks 202a-b. Multiple client devices 104 may travel to multiple locations and use different networks to access the cloud networks 202a-b.

[0040] In this embodiment, each service in services 212a-b includes one or more components / subsystems of an HTAP system (e.g., HTAP system 110). HTAP system 110 may include multiple different components (e.g., subsystems). For example, the HTAP system may include one or more of a transactional OLTP engine, an analytics OLAP engine, an underlying decoupled storage device, a metadata service, and / or an intelligent agent. Service 212a may include one or more (but not all) of the components of the HTAP system. Service 212b may include the remaining components of the HTAP system. Although... Figure 2 Two cloud networks and their respective services are shown in the document, but it should be understood that any number of cloud networks / services can be used to implement the HTAP system described herein.

[0041] Figure 3a An exemplary architecture 300 of an HTAP system (e.g., HTAP system 110) is illustrated. Architecture 300 includes several different components (e.g., subsystems). Subsystems include a transactional OLTP engine 304, an analytics OLAP engine 308, an underlying decoupled storage device 310, a metadata service 306, and / or a broker 302.

[0042] OLTP Engine 304 can be any OLTP engine configured to handle OLTP transactions. OLTP transactions can include DML and point lookup queries. OLTP Engine 304 can adopt a model that decouples the compute engine from the underlying shared / cloud storage (e.g., similar to AWS Aurora). OLTP Engine 304 provides a variety of features, including but not limited to ACID transaction support, row storage, write-ahead logging (WAL), and log replication. An exemplary architecture of OLTP Engine 304 will be described below regarding... Figure 4 To describe in more detail.

[0043] OLAP Engine 308 can be any OLAP engine configured to handle analytical queries, such as complex analytical queries. For example, OLAP Engine 308 can be an established open-source query engine (e.g., FlinkSQL). The following is about... Figure 5 An exemplary architecture for OLAP Engine 308 is described in more detail.

[0044] Storage device subsystem 310 can be decoupled from OLTP engine 304 and OLAP engine 308. Storage device 310 can persist user data in a first format (e.g., row format) for consumption by OLTP engine 304, and persist the same user data in a second format (e.g., mixed row + column format) for consumption by OLAP engine 308. This significantly reduces interference between OLAP and OLTP workloads. The following is about... Figure 6 An exemplary architecture of the storage device subsystem 310 is described in more detail.

[0045] Metadata service 306 can be configured to extract metadata from events (e.g., DDL) generated by certain resources (e.g., user input via OLTP engine 308), generate metadata versions, align them with the order of the DML, make them globally available, and persist them. Metadata service 306 can generate copies of the metadata versions for high availability. Metadata service 306 can extract metadata from DDL received by the OLTP engine (its version aligned with the same LSN system used for DML), persist it to a dedicated database, and push / pull it from FlinkSQL and AP storage unit servers.

[0046] Proxy 302 can be configured to connect clients to heterogeneous OLTP engine 304 and / or OLAP engine 308 through a common proxy layer. Proxy 302 provides a single, unified API (default: ANSL SQL plus some common OLAP extensions) to users / clients; that is, the underlying system details are transparent to the client if the client connects through the proxy. Various APIs (e.g., ANSI SQL, JDBC, ODBC, etc.) can be utilized based on user needs. Utilizing a single, unified API reduces the workload for users when using the system. Proxy 302 can automatically dispatch different client requests / queries to different engines (e.g., OLTP engine 304 or OLAP engine 308) based on the nature of the request. For example, complex OLAP queries will be routed to OLAP engine 308, while DML, DDL, and point lookup queries may be routed to OLTP engine 304.

[0047] The components of architecture 300 can be modularized and configured to be decoupled from each other. Therefore, components can be easily changed without modifying the entire architecture. The modular design also makes horizontal scaling of each component easier. For example, one or more of the OLTP engine 304, OLAP engine 308, storage device 310, metadata service 306, and / or agent 302 can be decoupled from the remaining components and replaced with alternative components (e.g., similar established modules or subsystems). For example, the current OLTP engine 304 may be MySQL, but it can be easily swapped to any other established OLTP engine, such as PostgreSQL. Similarly, the current OLAP engine 308 may be FlinkSQL, but it can be easily switched to any other established OLAP engine, such as Presto or Spark SQL. Storage device 310, metadata service 306, and / or agent 302 can similarly be replaced with alternative components.

[0048] Figure 3b An exemplary architecture 301 is illustrated for an HTAP system (e.g., HTAP system 110) where one or more components of architecture 300 are decoupled from the remaining components and replaced with alternative components. Architecture 301 includes multiple different components (e.g., subsystems). Subsystems include a transactional OLTP engine 314, an analytics OLAP engine 312, an underlying decoupled storage device 310, a metadata service 306, and / or a broker 302. Figure 3a The OLTP engine 304 was replaced by the OLTP engine 314. Similarly, Figure 3a The OLAP engine 308 is replaced by the OLAP engine 312. Although the storage device 310, metadata service 306, and agent 302 remain unchanged, in other embodiments, each of these components may be similarly replaced with an alternative component.

[0049] Figure 4An example system 400 for online transaction processing is illustrated. The OLTP system 400 includes an OLTP engine 402 that receives data or queries 401 from client devices (e.g., client devices 104a-104d). The OLTP engine 402 may utilize a model that decouples the computing engine from the underlying shared / cloud storage (similar to AWS Aurora). The OLTP engine 402 provides features such as ACID transaction support, row storage, write-ahead logging (WAL), and log replication. For example, the OLTP engine 402 may capture data in real-time or near real-time. The OLTP system 400 includes a data warehouse 403. The data warehouse 403 stores the data captured by the OLTP engine 402. The data warehouse 403 is accessible by an OLAP engine, allowing the OLAP engine to process the stored data in response to analytical queries. The OLTP engine 402 may be configured to be coupled to and / or decoupled from a larger HTAP system (e.g., HTAP system 110).

[0050] OLTP engine 402 can be configured to process at least some of the received user data in a specific format (e.g., row-based format). For example, upon receiving an OLTP query, OLTP engine 402 can be configured to process at least some of the received user data in response to that query. By way of example and not limitation, OLTP queries may include queries that retrieve specific data items, queries that filter received data to find specific data items / data item descriptions, and / or queries that filter received data to identify specific subsets of the received data. For example, queries may be DML queries and / or point lookup queries.

[0051] Figure 5 An example system 500 for online analytical processing is shown. The OLAP system 500 includes an OLAP engine 503, which receives data captured by an OLTP system. For example, the OLAP engine 503 can retrieve data stored above regarding... Figure 4 The data described is in data warehouse 403. The data received by OLAP engine 503 may be in a specific format readable by OLAP engine 503 (e.g., a mixed row + column format). OLAP engine 503 may perform analysis on at least one portion of the data in the specific format in response to a received OLAP query.

[0052] For example, OLAP Engine 503 can perform analytical operations on at least one part of the data. OLAP consists of three basic analytical operations: merge (roll-up), drill-down, and slicing and dicing. Merging involves the aggregation of data that can be accumulated and computed across one or more dimensions. For example, all sales offices are aggregated into a sales department or sales division to predict sales trends. In contrast, drill-down is a technique that allows users to browse details. For example, a user can view sales by individual products that make up regional sales. Slicing and dicing are features that allow users to extract (slice) a specific dataset from an OLAP cube and view (dic) the slice from different viewpoints. These viewpoints are sometimes referred to as dimensions (e.g., viewing the same sales by salesperson, by date, by customer, by product, or by region, etc.).

[0053] The processed results can be sent or forwarded to client device 504. OLAP engine 503 can be any OLAP engine capable of efficiently handling complex analytical queries, such as established open-source query engines (e.g., FlinkSQL). OLAP engine 503 can be configured to be coupled to and / or decoupled from a larger HTAP system (e.g., HTAP system 110).

[0054] Figure 6 An example system 600 for a data storage device according to this disclosure is shown. The data storage device system 600 can persist user data in one format (e.g., row format) for consumption by an OLTP engine, while simultaneously persisting the same user data in a different format (e.g., a mixed row + column format) for consumption by an OLAP engine. This significantly reduces interference between OLAP and OLTP workloads. For high availability, multiple copies of the data (e.g., three copies) can be maintained for both data formats.

[0055] The data storage system 600 can be used as a unified storage layer. However, the architecture of the data storage system 600 can be divided into two parts: the TP part and the AP part. Transactional DML and DDL received by the OLTP engine can be presented as physical / redo logs (with information from the underlying storage devices) and logical logs in the log data warehouse 602. These logs can be persisted in the TP part of the storage device. The physical logs can then be replicated and distributed to other storage devices by the log distributor 604 and replayed to build data pages. The data in the pages can be organized in a row format and stored in the row data warehouse 606. The data stored in the row data warehouse 606 can be used by the OLTP engine for simple queries, including point lookup queries. The logical logs can also be replicated and distributed to the AP part of the storage device by the log distributor 604.

[0056] Each user table in the AP component of the storage device can be partitioned based on a partitioning scheme defined when the table is created. Each partition 607 can reside on a physical storage node (e.g., a server) and can maintain several copies of the data for high availability. Each partition 607 can be further divided into an in-memory incremental data warehouse 608 and an on-disk basic data warehouse 610. The logical log (such as a MySQL binary log) for committed DML transactions can be continuously dispatched from the OLTP engine to the user table partitions on each AP storage node. As an example, updates can be modeled as inserts after deletes. Therefore, in the logical log, only inserts and deletes may exist, without updates.

[0057] Logical logs arriving at each AP storage node can be sorted, persisted, and then sequentially applied to the in-memory incremental data warehouse 608 of each partition 607. The incremental data warehouse 608 stores data in a row format, sorted by its log sequence number (LSN, a sequence number / version indicating the order of data operations). Data within the incremental data warehouse 608 can be periodically flushed to the base data warehouse 610, for example, when its size exceeds a predefined threshold or after a fixed time interval. After flushing, the memory occupied by the flushed data can be garbage collected.

[0058] Data in the base data warehouse 610 can be organized in a columnar format for better performance of analytical workloads and persisted in a local file system (in the current implementation). However, it should be understood that this architecture can work with any underlying storage device approach. Data in the base data warehouse 610 may not be updated in-place, therefore a deletion bitmap marking deleted rows can be maintained. As more and more data units are deleted and overlapped with flushes, compressing and rearranging the internal data can benefit storage and potential query performance, so AP data units and cleaning up redundant data can be compressed periodically. Files in the base data warehouse 610 can be versioned, which is the largest LSN in the file when the file is flushed. The LSN may not be stored in the base data warehouse 610, which results in all files in the base data warehouse 610 having the same version (i.e., the last flushed LSN). This provides several advantages. For example, it saves storage space, makes deletion more efficient by using a deletion bitmap, and makes scans faster when comparison with LSN is not required.

[0059] Figure 7 An exemplary architecture 700 for an HTAP system according to this disclosure is shown. Architecture 700 may be as described above. Figures 3a-3b A more detailed view of the described HTAP system architectures 300 and 301 is provided. Architecture 700 includes several different components (e.g., subsystems). Subsystems include those referenced above. Figure 4The OLTP engine 402 described above is referenced. Figure 5 The OLAP engine described above (reference 503) Figure 6 The data storage device system 600, metadata service 704, and / or agent 702 are described.

[0060] As described above, OLTP engine 402 can receive user data, such as user data from user devices 701a-n. OLTP engine 402 can utilize a model that decouples the computing engine from the underlying shared / cloud storage device (similar to AWS Aurora). OLTP engine 402 can provide functionality such as ACID transaction support, row storage, write-ahead log (WAL), and log replication. For example, OLTP engine 402 can capture user data in real-time or near real-time. OLTP system 400 includes a data warehouse 403. Data warehouse 403 can store the data captured by OLTP engine 402. Data warehouse 403 can be accessed by OLAP engine, allowing OLAP engine to process the stored data in response to analytical queries.

[0061] OLTP engine 402 can be configured to process at least some of the received user data in a specific format (e.g., row-based format). For example, upon receiving an OLTP query, OLTP engine 402 can be configured to process at least some of the received user data in response to that query. By way of example and not limitation, OLTP queries may include queries that retrieve specific data items, queries that filter received data to find specific data items / data item descriptions, and / or queries that filter received data to identify specific subsets of the received data. For example, queries may be DML queries and / or point lookup queries.

[0062] OLTP engine 402 can be configured to be coupled to and / or decoupled from the larger HTAP system 700. If OLTP engine 402 is decoupled from the larger HTAP system 700, then OLTP engine 402 can be replaced by an alternative OLTP engine (e.g., a similar established OLTP engine). For example, OLTP engine 402 can be MySQL, but it can be easily swapped to any other established OLTP engine, such as PostgreSQL.

[0063] User equipment 701a-n may include any type of computing device, such as mobile devices, tablets, laptops, desktop computers, smart TVs or other smart devices (e.g., smartwatches, smart speakers, smart glasses, smart helmets), gaming devices, set-top boxes, digital streaming devices, robots, and / or similar items. User equipment 701a-n may be associated with one or more users. A single user may use one or more of user equipment 701a-n to access a cloud network including OLTP engine 402. User equipment 701a-n may travel to multiple locations and use different networks to access the cloud network including OLTP engine 402.

[0064] As described above, OLAP engine 503 can receive data captured by OLTP engines, such as OLTP engine 402. For example, OLAP engine 503 can retrieve data stored in data warehouse 403. The data received by OLAP engine 503 may be in a specific format readable by OLAP engine 503 (e.g., a mixed row + column format). OLAP engine 503 can perform analysis on at least one portion of the data in the specific format in response to receiving an OLAP query.

[0065] For example, the OLAP engine 503 can perform analytical operations on at least one portion of the data. OLAP consists of three basic analytical operations: merging (roll-up), drill-down, and slicing and dicing. Merging involves the aggregation of data that can be accumulated and computed across one or more dimensions. For example, all sales offices are aggregated into a sales department or sales division to predict sales trends. In contrast, drill-down is a technique that allows users to browse details. For example, a user can view sales by individual products that make up regional sales. Slicing and dicing are features from which users can extract (slice) OLAP cubes and view (dic) slices from different viewpoints. These viewpoints are sometimes referred to as dimensions (e.g., viewing the same sales by salesperson, by date, by customer, by product, or by region, etc.). The results of the processing can be sent or forwarded to the client device 504.

[0066] OLAP Engine 503 can be any OLAP engine capable of efficiently handling complex analytical queries, such as established open-source query engines (e.g., FlinkSQL). OLAP Engine 503 can be configured to be coupled to and / or decoupled from the larger HTAP system 700. If OLAP Engine 503 is decoupled from the larger HTAP system 700, it can be replaced by an alternative OLAP engine (e.g., a similar established OLAP engine). For example, OLAP Engine 503 could be FlinkSQL, but it can be easily swapped to any other established OLAP engine, such as Presto or Spark SQL.

[0067] The storage device subsystem 600 can be decoupled from the OLTP engine 402 and the OLAP engine 503. Also as described above, the data storage device system 600 can persist user data in one format (e.g., row format) for consumption by the OLTP engine (e.g., OLTP engine 402), while persisting the same user data in a different format (e.g., mixed row + column format) for consumption by the OLAP engine (e.g., OLAP engine 503). This significantly reduces interference between OLAP and OLTP workloads. For high availability, multiple copies of the data (e.g., three copies) can be maintained for both data formats.

[0068] The data storage device system 600 can be used as a unified storage layer. However, the architecture of the data storage device system 600 can be divided into two components: the TP component and the AP component. Transactional DML and DDL received by the OLTP engine can be presented as physical / redo logs (with information from the underlying storage device) and logical logs in the log data warehouse 602. These logs can be persisted in the TP component of the storage device. The physical logs can then be replicated and distributed to other storage devices by the log distributor 604 and replayed to build data pages. The data in the pages can be organized in a row format and stored in the row data warehouse 606. The data stored in the row data warehouse 606 can be used by the OLTP engine for simple queries, including point lookup queries. The logical logs can also be replicated and distributed to the AP component of the storage device by the log distributor 604.

[0069] Each user table in the AP component of the storage device can be partitioned based on a partitioning scheme defined when the table is created. Each partition 607 can reside on a physical storage node (e.g., a server) and can maintain several copies of the data for high availability. Each partition 607 can be further divided into an in-memory incremental data warehouse 608 and an on-disk basic data warehouse 610. The logical log (such as a MySQL binary log) for committed DML transactions can be continuously dispatched from the OLTP engine to the user table partitions on each AP storage node. As an example, updates can be modeled as inserts after deletes. Therefore, in the logical log, only inserts and deletes may exist, without updates.

[0070] Logical logs arriving at each AP storage node can be sorted, persisted, and then sequentially applied to the in-memory incremental data warehouse 608 of each partition 607. The incremental data warehouse 608 stores data in a row format, sorted by its log sequence number (LSN, a sequence number / version indicating the order of data operations). Data within the incremental data warehouse 608 can be periodically flushed to the base data warehouse 610, for example, when its size exceeds a predefined threshold or after a fixed time interval. After flushing, the memory occupied by the flushed data can be garbage collected.

[0071] Data in the base data warehouse 610 can be organized in a columnar format for better performance of analytical workloads and persisted in a local file system (in the current implementation). However, it should be understood that this architecture can work with any underlying storage device approach. Data in the base data warehouse 610 may not be updated in-place, therefore a deletion bitmap marking deleted rows can be maintained. As more and more data units are deleted and overlapped with flushes, compressing and rearranging the internal data can benefit storage and potential query performance, so AP data units and cleaning up redundant data can be compressed periodically. Files in the base data warehouse 610 can be versioned, which is the largest LSN in the file when the file is flushed. The LSN may not be stored in the base data warehouse 610, which results in all files in the base data warehouse 610 having the same version (i.e., the last flushed LSN). This provides several advantages. For example, it saves storage space, makes deletion more efficient by using a deletion bitmap, and makes scans faster when comparison with LSN is not required.

[0072] The storage device subsystem 600 can be configured to be coupled to, for example Figure 7 The larger HTAP system 700 shown is decoupled from it. If the storage device subsystem 600 is decoupled from the larger HTAP system 700, the storage device subsystem 600 can be replaced by an alternative storage device subsystem (e.g., a similar established storage device subsystem).

[0073] Metadata service 704 can be configured to align metadata between OLTP engine 402 and OLAP engine 503. Metadata service 704 can extract metadata from events (e.g., DDL) generated by certain resources (e.g., user input via OLTP engine 308), generate metadata versions, align them with the order of the DML, make them globally available, and persist them. Metadata service 306 can generate copies of the metadata versions for high availability. Metadata service 306 can extract metadata from DDL received by the OLTP engine (version aligned with the same LSN system used for the DML), persist it to a dedicated database, and push / pull it by FlinkSQL and AP storage unit servers.

[0074] Metadata service 704 can be configured to be coupled to and / or decoupled from the larger HTAP system 700. If metadata service 704 is decoupled from the larger HTAP system 700, metadata service 704 can be replaced by an alternative metadata service (e.g., a similar established metadata service).

[0075] Agent 702 can be configured to connect user devices 701a-n to OLTP engine 402 and / or OLAP engine 503 through a common proxy layer. Agent 702 can be a smart agent. Agent 702 can provide a single unified API (default: ANSL SQL plus some common OLAP extensions) to users / clients, meaning that the underlying system details are transparent to the client if the client connects through the agent. Various APIs (e.g., ANSI SQL, JDBC, ODBC, etc.) can be utilized based on user needs. Utilizing a single unified API reduces the workload for users when using the system. Agent 702 can have the ability to automatically dispatch different client requests / queries to different engines (e.g., OLTP engine 402 or OLAP engine 503) based on the nature of the request. For example, complex OLAP queries will be directed to OLAP engine 503, while DML, DDL, and point lookup queries will be directed to OLTP engine 402.

[0076] Agent 702 can be configured to be coupled to and / or decoupled from the larger HTAP system 700. If agent 702 is decoupled from the larger HTAP system 700, agent 702 can be replaced by an alternative common agent (e.g., a similar established agent).

[0077] Figure 8 The diagram illustrates what can be achieved by an HTAP system, such as... Figure 7 The example process 800 executed by the HTAP system 700 is shown. The HTAP system executes process 800 to receive and / or organize user data in a manner that can be utilized by both the OLTP and OLAP subsystems. Although in Figure 8 The operations are depicted as a sequence of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder, or modify the depicted operations.

[0078] In 802, data can be captured in real time (or near real time) by a first computing system. The first computing system can be, for example, an OLTP engine (e.g., OLTP engine 402). Since OLTP workloads inherently perform better with row-based data formats, while OLAP workloads prefer columnar data formats, using a single format for both workloads means that the performance of the HTAP system will be affected.

[0079] To prevent this performance loss, user data can be stored in different formats for use by the OLTP engine and the OLAP engine. For example, user data can be saved in a first format (e.g., row-based format) for the OLTP engine and in a second format (e.g., column-based format) for the OLAP engine for efficient query processing. At 804, data can be organized in the first format in a first component of the storage device subsystem for use by the first processing engine.

[0080] To organize data in a first format within the first component of the storage device subsystem, transactional DML and DDL received by the OLTP engine can be presented as physical / redo logs (containing information about the underlying storage device) and logical logs in a log data warehouse (e.g., log data warehouse 602). These logs can be persisted in the first component of the storage device (e.g., the TP component). The physical logs can then be copied and distributed to other storage devices by a log distributor (e.g., log distributor 604) and replayed to construct data pages. Data in the pages can be organized in a row format and stored in a row data warehouse (e.g., row data warehouse 606). The data stored in the row data warehouse can be used by the OLTP engine for simple queries, including point lookup queries.

[0081] The logical log can also be replicated and distributed by the log distributor to the second part of the storage subsystem (e.g., the AP part). In 806, data can be propagated to the second part of the storage subsystem. Each user table in the second part of the storage subsystem can be partitioned based on a partitioning scheme defined at the time of table creation. Each partition (e.g., partition 607) can reside on a physical storage node (e.g., a server) and can maintain several copies of the data for high availability. Each partition can be further divided into an in-memory incremental data warehouse (e.g., incremental data warehouse 608) and an on-disk basic data warehouse (e.g., basic data warehouse 610). The logical log (such as the MySQL binary log) for committed DML transactions can be continuously dispatched from the OLTP engine to the user table partitions on each AP storage node. As an example, updates can be modeled as inserts after deletions. Therefore, in the logical log, only inserts and deletions may exist, without updates.

[0082] In 808, data can be organized in a second format within a second component of the storage subsystem for use by a second processing engine (e.g., an OLAP subsystem). Logical logs arriving at each AP storage node can be sorted, persisted, and then sequentially applied to the in-memory incremental data warehouse for each partition. The incremental data warehouse can store data in a row format sorted by log sequence number (LSN, a sequence number / version indicating the order of data operations). Data within the incremental data warehouse can be periodically flushed to the base data warehouse, for example, when its size exceeds a predefined threshold or after a fixed time interval. After flushing, the memory occupied by the flushed data can be garbage collected.

[0083] Data in the base data warehouse can be organized in a columnar format for better performance of analytical workloads and persisted in a local file system (in the current implementation). However, it should be understood that this architecture can work with any underlying storage approach. Data in the base data warehouse may not be updated in-place, so a deletion bitmap marking deleted rows can be maintained. As more and more data units are deleted and overlapped during flushing, compressing and rearranging the internal data can benefit storage and potential query performance; therefore, AP data units and cleaning up redundant data can be compressed periodically. Files in the base data warehouse can be versioned, which is the largest LSN in the file at the time of flushing. The LSN may not be stored in the base data warehouse, which results in all files in the base data warehouse having the same version (i.e., the last flushed LSN). This provides several advantages. For example, it saves storage space, makes deletion more efficient by using a deletion bitmap, and makes scans faster when comparison with LSN is not required.

[0084] Figure 9 The illustration depicts an example process 900 that can be executed by a centralized metadata service of an HTAP system. For example, process 900 can be executed by, for instance, a centralized metadata service of an HTAP system. Figure 7 The metadata service 704 of the HTAP system 700 is executed. The HTAP system executable process 900 aligns the metadata to integrate the OLTP subsystem and OLAP subsystem within the HTAP system. Although in Figure 9 The operations are depicted as a sequence of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder, or modify the depicted operations.

[0085] In section 902, metadata can be extracted from data captured by a first processing engine (e.g., an OLTP engine). Metadata can be extracted from events (e.g., DDL) generated by some resource (e.g., user input via the OLTP engine). For example, metadata can be extracted from DDL received by the OLTP engine. In section 904, a metadata version can be generated based on the extracted metadata. In section 906, the metadata version can be aligned with the order of the Data Manipulation Language (DML). For example, the metadata version can be aligned with the same LSN system used for DML. The metadata version can be globally available and persistent. For example, the metadata version can be persisted to a dedicated database and pushed / pulled by FlinkSQL and AP storage unit servers. A copy of the metadata version can be generated for high availability.

[0086] Figure 10 An example hybrid transactional / analytic process 1000 according to this disclosure is illustrated. Process 1000 can be executed by an HTAP system (e.g., HTAP system 110). The HTAP system can execute process 1000 to receive and / or organize user data in a manner that can be utilized by both OLTP and OLAP subsystems. Although in Figure 10 The operations are depicted as a sequence of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder, or modify the depicted operations.

[0087] The common agent layer can be configured to connect clients to the OLTP and / or OLAP engines. In 1002, multiple client computing devices can connect to the first and second processing engines via the common agent layer. The agent provides a single, unified API to users / clients (default: ANSI SQL plus some common OLAP extensions), meaning that the underlying system details are transparent to the client if they connect through the agent. Various APIs (e.g., ANSI SQL, JDBC, ODBC, etc.) can be utilized based on user needs. Using a single, unified API reduces the workload for users when using the system.

[0088] At 1004, data can be captured in real-time (or near real-time) by a first computing system. The first computing system can be, for example, an OLTP engine (e.g., OLTP engine 402). Since OLTP workloads inherently perform better with row-based data formats, while OLAP workloads prefer column-based data formats, using a single format for both workloads would negatively impact the performance of the HTAP system. To prevent this performance loss, user data can be stored in different formats for use by the OLTP and OLAP engines. At 1006, captured data can be organized in a first format in a first component of the storage subsystem for use by the first processing engine, and in a second format in a second component of the storage subsystem for use by the second processing engine (e.g., OLAP engine 503). For example, user data can be stored in a first format (e.g., row-based format) for use by the OLTP engine and in a second format (e.g., column-based format) for use by the OLAP engine for efficient query processing.

[0089] Metadata can be extracted from captured data. Metadata can be extracted from events (e.g., DDL) generated by certain resources (e.g., user input via an OLTP engine). For example, metadata can be extracted from DDL received by an OLTP engine. A metadata version can be generated based on the extracted metadata. At 1008, metadata associated with the captured data can be aligned for integration with a first processing engine and a second processing engine. For example, the metadata version can be aligned with the order of the Data Manipulation Language (DML). The metadata version can be aligned with the same LSN system used for DML. The metadata version can be globally available and persistent. For example, the metadata version can be persisted to a dedicated database and pushed / pulled by FlinkSQL and AP storage transpose servers. A copy of the metadata version can be generated for high availability.

[0090] The agent can automatically dispatch different client requests / queries to different engines (e.g., OLTP or OLAP engines) based on the nature of the request. In 1010, the common agent layer can dispatch queries to a first or second processing engine based on the type of query received from multiple computing devices. For example, complex OLAP queries will be directed to the OLAP engine, while DML, DDL, and point lookup queries may be directed to the OLTP engine.

[0091] The agent may direct at least a portion of the data to the first processing engine. At 1012, in response to receiving an OLTP query, at least a portion of the data in a first format may be processed by the first processing engine. The first processing engine may be configured to process at least some of the received user data in the first format (e.g., a row-based format). For example, upon receiving an OLTP query, the OLTP engine may be configured to process at least some of the received user data in response to the OLTP query. By way of example and not limitation, an OLTP query may include a query to retrieve a specific data item, a query to filter received data to find a specific data item / data item description, and / or a query to filter received data to identify a specific subset of the received data. For example, the query may be a DML query and / or a point lookup query. The result of the processing may be sent or forwarded to the client device that generated the query.

[0092] An agent can direct at least a portion of the data to a second processing engine. In step 1014, in response to receiving an OLAP query, the second processing engine performs analysis on at least a portion of the data in a second format. The second computing system can perform analytical operations on at least a portion of the data. OLAP consists of three basic analytical operations: merging (roll-up), drilling down, and slicing and dicing. Merging involves the aggregation of data that can be accumulated and computed across one or more dimensions. For example, all sales offices are aggregated into a sales department or sales division to predict sales trends. In contrast, drilling down is a technique that allows users to browse details. For example, a user can view sales by individual products that constitute regional sales. Slicing and dicing are features from which users can extract (slice) the OLAP cube and view (dic) the slices from different viewpoints. These viewpoints are sometimes referred to as dimensions (e.g., viewing the same sales by salesperson, by date, by customer, by product, or by region, etc.). The results of the processing can be sent or forwarded to the client device that generated the query.

[0093] Figure 11 The illustration shows computing devices that can be used in various fields, such as Figure 1 The services, networks, modules, and / or devices shown. About Figure 1 In the example architecture, each of cloud network 102, network 120, client devices 104a-d, service 112, HTAP system 110, and / or node 118 can be provided by Figure 11 The computing device 1100 is used to implement this. Figure 11The computer architecture shown illustrates conventional server computers, workstations, desktop computers, laptops, tablet devices, network devices, PDAs, e-readers, digital cellular phones, or other computing nodes, and can be used to perform any aspect of the computer described herein, such as implementing the methods described herein.

[0094] The computing device 1100 may include a substrate or “motherboard,” which is a printed circuit board that can connect multiple components or devices via a system bus or other electrical communication path. One or more central processing units (CPUs) 1104 may operate in conjunction with a chipset 1106. The CPUs (multiple) 1104 may be standard programmable processors that perform arithmetic and logic operations required to perform the operation of the computing device 1100.

[0095] Multiple CPUs 1104 can perform necessary operations by manipulating switching elements to transition from one discrete physical state to another, these switching elements distinguishing and changing these states. Switching elements typically include electronic circuitry, such as flip-flops, that maintains one of two binary states, and electronic circuitry, such as logic gates, that provides an output state based on a logical combination of the states of one or more other switching elements. These basic switching elements can be combined to create more complex logic circuits, including registers, adder-subtractor units, arithmetic logic units, floating-point units, and similar items.

[0096] The (multiple) CPUs 1104 may be expanded or replaced by other processing units such as (multiple) GPUs 1105. The (multiple) GPUs 1105 may include processing units dedicated to, but not limited to, highly parallel computing, such as graphics and other visualization-related processing.

[0097] Chipset 1106 provides an interface between CPU(s) 1104 and the remaining components and devices on the substrate. Chipset 1106 provides an interface to random access memory (RAM) 1108, which serves as main memory in computing device 1100. Chipset 1106 may also provide an interface to computer-readable storage media, such as read-only memory (ROM) 1120 or non-volatile RAM (NVRAM) (not shown), for storing basic routines that help start computing device 1100 and transfer information between various components and devices. ROM 1120 or NVRAM may also store other software components necessary for the operation of computing device 1100 according to the aspects described herein.

[0098] Computing device 1100 can operate in a networked environment via a local area network (LAN) using logical connections to remote computing nodes and computer systems. Chipset 1106 may include functionality for providing network connectivity via a network interface controller (NIC) 1122 (such as a Gigabit Ethernet adapter). NIC 1122 enables computing device 1100 to connect to other computing nodes via network 1116. It should be understood that multiple NICs 1122 may be present in computing device 1100 to connect the computing device to other types of networks and remote computer systems.

[0099] Computing device 1100 can be connected to mass storage device 1128, which provides non-volatile storage for the computer. Mass storage device 1128 can store system programs, application programs, other program modules, and data, which have been described in more detail herein. Mass storage device 1128 can be connected to computing device 1100 via storage controller 1124, which is connected to chipset 1106. Mass storage device 1128 may consist of one or more physical storage units. Mass storage device 1128 may include management component 1010. Storage controller 1124 can interface with physical storage units via Serial Attached SCSI (SAS) interface, Serial Advanced Technology Attachment (SATA) interface, Fibre Channel (FC) interface, or other types of interfaces used for physical connection and data transfer between the computer and physical storage units.

[0100] The computing device 1100 can store data on the mass storage device 1128 by transforming the physical state of the physical storage units to reflect the stored information. The specific transformation of the physical state may depend on various factors and different implementations described herein. Examples of these factors may include, but are not limited to, the technology used to implement the physical storage units and whether the mass storage device 1128 is characterized as a primary storage device or a secondary storage device and the like.

[0101] For example, computing device 1100 may issue instructions via storage controller 1124 to store information in mass storage device 1128 to alter the magnetic properties of a specific location within a disk drive unit, the reflection or refraction properties of a specific location within an optical storage unit, or the electrical properties of a specific capacitor, transistor, or other discrete component in a solid-state storage unit. Other transformations of the physical medium are possible without departing from the scope and spirit of this description; the foregoing examples are provided merely for the purpose of illustration. Computing device 1100 may also read information from mass storage device 1128 by detecting the physical state or characteristics of one or more specific locations within the physical storage unit.

[0102] In addition to the aforementioned high-capacity storage device 1128, the computing device 1100 may also access other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. Those skilled in the art will understand that a computer-readable storage medium can be any available medium that provides storage for non-transitory data and is accessible by the computing device 1100.

[0103] By way of example and not limitation, computer-readable storage media may include volatile and non-volatile, transient and non-transitory computer-readable storage media implemented in any method or technology, as well as removable and non-removable media. Computer-readable storage media include, but are not limited to, RAM, ROM, erasable programmable ROM (“EPROM”), electrically erasable programmable ROM (“EEPROM”), flash memory or other solid-state memory technologies, optical disc ROM (“CD-ROM”), digital versatile disc (“DVD”), high-definition DVD (“HD-DVD”), BLU-RAY or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices, other magnetic storage devices, or any other medium that can be used to store desired information in a non-transitory manner.

[0104] High-capacity storage devices, such as Figure 11 The mass storage device 1128 shown can store an operating system used to control the operation of the computing device 1100. The operating system may include a version of the Linux operating system. The operating system may include a version of the Windows Server operating system from Microsoft Corporation. Depending on another aspect, the operating system may include a version of the UNIX operating system. Various mobile phone operating systems, such as iOS and Android, may also be used. It should be understood that other operating systems may also be utilized. The mass storage device 1128 can store other systems, applications, and data used by the computing device 1100.

[0105] Mass storage device 1128 or other computer-readable storage media may also be encoded with computer-executable instructions that, when loaded into computing device 1100, transform the computing device from a general-purpose computing system into a special-purpose computer capable of implementing the aspects described herein. As described above, these computer-executable instructions transform computing device 1100 by specifying how CPU(one or more) 1104 transitions between states. Computing device 1100 has access to computer-readable storage media storing computer-executable instructions that, when executed by computing device 1100, perform the methods described herein.

[0106] Computing devices, such as Figure 11The computing device 1100 shown may also include an input / output controller 1132 for receiving and processing input from a plurality of input devices, such as a keyboard, mouse, touchpad, touchscreen, electronic pen, or other types of input devices. Similarly, the input / output controller 1132 may provide output to a display, such as a computer monitor, flat panel display, digital projector, printer, plotter, or other types of output device. It should be understood that the computing device 1100 may not include... Figure 11 All of the components shown may include Figure 11 Other components not explicitly shown in the document, or those that can be used with Figure 11 The architecture shown is completely different.

[0107] As described in this article, a computing device can be a physical computing device, such as... Figure 11 The computing device 1100. The computing node may also include virtual machine host processes and one or more virtual machine instances. Computer-executable instructions may be indirectly executed by the physical hardware of the computing device by interpreting and / or executing instructions stored and executed in the context of the virtual machine.

[0108] It should be understood that the method and system are not limited to a particular method, component, or implementation. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

[0109] As used in the specification and appended claims, unless the context clearly specifies otherwise, the singular forms “a,” “an,” and “the” include plural indicators. A range may be expressed herein as from “about” a particular value and / or to “about” another particular value. When expressing such ranges, another embodiment includes from one particular value and / or to another particular value. Similarly, when a value is expressed as an approximation using the antecedent “about,” it will be understood that the particular value forms another embodiment. It should also be understood that each endpoint in the range is significant both relative to and independent of the other endpoint.

[0110] "Optional" or "optionally" means that the event or situation described below may or may not occur, and the description includes instances where the event or situation occurs and instances where it does not occur.

[0111] Throughout the description and claims of this specification, the word "comprise" and its variations, such as "comprising" and "comprises," mean "including but not limited to," and do not exclude, for example, other components, integrals, or steps. "Exemplary" means "an example of" and is not intended to convey indications of preferred or ideal embodiments. "Such as" is not used in a limiting sense but for interpretive purposes.

[0112] Components that can be used to perform the described methods and systems are described. When describing combinations, subsets, interactions, groups, etc., of these components, it should be understood that while no specific reference is explicitly made to each of the various individual and collective combinations and arrangements of these components, each is specifically contemplated and described herein for all methods and systems. This applies to all aspects of this application, including but not limited to the operations in the described methods. Therefore, if various additional operations are available, it should be understood that each of these additional operations can be performed using any particular embodiment or combination of embodiments of the described methods.

[0113] The method and system can be more readily understood by referring to the detailed description of the following preferred embodiments and the examples included therein, as well as the accompanying drawings and their descriptions.

[0114] As those skilled in the art will recognize, the method and system may take the form of an all-hardware embodiment, an all-software embodiment, or a combination of software and hardware. Furthermore, the method and system may take the form of a computer program product on a computer-readable storage medium having computer-readable program instructions (e.g., computer software) embodied therein. More specifically, the method and system may take the form of network-implemented computer software. Any suitable computer-readable storage medium may be utilized, including hard disks, CD-ROMs, optical storage devices, or magnetic storage devices.

[0115] The following describes embodiments of methods and systems with reference to block diagrams and flowcharts of methods, systems, apparatuses, and computer program products. It should be understood that each block in the block diagrams and flowcharts, as well as combinations of blocks in the block diagrams and flowcharts, can be implemented by computer program instructions. These computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the computer or other programmable data processing apparatus, create means for performing the functions specified in one or more flowchart blocks.

[0116] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture, including computer-readable instructions for implementing the functions specified in one or more flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus, thereby producing a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowchart blocks.

[0117] The various features and processes described above can be used independently of each other or in combination in various ways. All possible combinations and sub-combinations fall within the scope of this disclosure. Furthermore, in some embodiments, certain method or process blocks may be omitted. The methods and processes described herein are not limited to any particular sequence, and the blocks or states associated with them may be performed in other suitable sequences. For example, the described blocks or states may be performed in a different order than specifically described, or multiple blocks or states may be combined in a single block or state. Example blocks or states may be performed serially, in parallel, or in some other manner. Blocks or states may be added to or removed from the described exemplary embodiments. The example systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the described example embodiments.

[0118] It should also be understood that various items are shown to be stored in memory or storage devices during use, and these items, or portions thereof, may be transferred between memory and other storage devices for memory management and data integrity purposes. Alternatively, in other embodiments, some or all of the software modules and / or systems may be implemented in memory on another device and communicate with the illustrated computing system via inter-computer communication. Furthermore, in some embodiments, some or all of the systems and / or modules may be implemented or provided in other ways, such as at least in part as firmware and / or hardware, including, but not limited to, one or more application-specific integrated circuits (“ASICs”), standard integrated circuits, controllers (e.g., by implementing appropriate instructions, and including microcontrollers and / or embedded controllers), field-programmable gate arrays (“FPGAs”), complex programmable logic devices (“CPLDs”), etc. Some or all of the modules, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a computer-readable medium, such as a hard disk, memory, network, or portable media product, for retrieval by an appropriate device or via an appropriate connector. Systems, modules, and data structures can also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) over various computer-readable transmission media, including wireless and wired / cable-based media, and can take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, such computer program products may also take other forms. Therefore, the invention can be practiced with other computer system configurations.

[0119] Although the methods and systems have been described in conjunction with preferred embodiments and specific examples, they are not intended to limit the scope to the specific embodiments illustrated, as the embodiments herein are intended in all respects to be illustrative rather than restrictive.

[0120] Unless otherwise expressly stated, no method described herein shall be construed as requiring its operations to be performed in a particular order. Therefore, no order is intended to be inferred in any event where a method claim does not actually state the order in which its operations are performed, or where the claims or description do not otherwise specify that the operations will be limited to a particular order. This applies to any possible non-express basis of interpretation, including: logical questions concerning the arrangement of steps or the flow of operations; naive meanings derived from grammatical organization or punctuation; and the number or type of embodiments described in the specification.

[0121] It will be apparent to those skilled in the art that various modifications and variations may be made without departing from the scope or spirit of this disclosure. Other embodiments will be apparent to those skilled in the art in light of the specification and practice described herein. The specification and example drawings are to be considered exemplary only, and the true scope and spirit are indicated by the appended claims.

Claims

1. A hybrid data processing system comprising: at least one processor; at least one memory communicatively coupled to the at least one processor and comprising instructions that, when executed by the at least one processor, cause the system to perform operations comprising: capturing data in real-time by a first processing engine, the first processing engine configured to perform online transaction processing; organizing the data in a first component of a storage device in a first format for use by the first processing engine; propagating the data to a second component of the storage device; and organizing the data in the second component of the storage device in a second format for use by a second processing engine, the second processing engine configured to perform online analytical processing, wherein the first processing engine, the second processing engine, and the storage device are modularized and configured to be decoupled from one another; wherein the operations further comprise: extracting metadata from the data by a centralized metadata service; generating a metadata version based on the extracted metadata by the centralized metadata service; and aligning the metadata version with a sequence of a data manipulation language by the centralized metadata service.

2. The system of claim 1, further comprising: the centralized metadata service configured to align metadata for integration of the first processing engine and the second processing engine.

3. The system of claim 1, further comprising: a common broker layer configured to connect a plurality of client computing devices to the first processing engine and the second processing engine.

4. The system of claim 3, the operations further comprising: dispatching, by the common broker layer, a query to the first processing engine or the second processing engine based on a type of the query received from the plurality of computing devices.

5. The system of claim 1, the operations further comprising: processing, by the first processing engine, at least one portion of the data in the first format in response to receiving a query for online transaction processing.

6. The system of claim 1, the operations further comprising: performing, by the second processing engine, analysis on at least one portion of the data in the second format in response to receiving a query for online analytical processing.

7. The system of claim 1, the operations further comprising: generating a physical log and a logical log based on the data; replicating the physical log or the logical log for building data pages; and organizing data in the data pages in the first format for use by the first processing engine.

8. The system of claim 7, the operations further comprising: replicating the logical log and distributing to the second component of the storage device; and categorizing copies of the logical log into a plurality of partitions based on a predetermined partitioning scheme.

9. The system of claim 8, the operations further comprising: ​ ​ applying at least one portion of the copy of the logical log in order to an incremental data store of a corresponding partition of the plurality of partitions, wherein data in the incremental data store is stored in a row format and ordered based on a log sequence number associated with the at least one portion of the copy of the logical log; and flushing data from the incremental data store to a base data store based on one or more predetermined rules, wherein data in the base data store is organized in a column format for use by the second processing engine.

10. A hybrid data processing method, comprising: capturing data in real-time by a first processing engine, the first processing engine configured to perform online transaction processing; organizing the data in a first format in a first component of a storage device for use by the first processing engine; propagating the data to a second component of the storage device; organizing the data in a second format in the second component of the storage device for use by a second processing engine, the second processing engine configured to perform online analytical processing, wherein the first processing engine, the second processing engine, and the storage device are modularized and configured to be decoupled from one another; extracting metadata from the data by a centralized metadata service; generating a metadata version based on the extracted metadata by the centralized metadata service; and aligning the metadata version with a data manipulation language order by the centralized metadata service.

11. The method of claim 10, further comprising: aligning metadata for integration of the first processing engine and the second processing engine by the centralized metadata service.

12. The method of claim 10, further comprising: connecting a plurality of client computing devices to the first processing engine and the second processing engine by a common agent layer.

13. The method of claim 12, further comprising: dispatching a query to the first processing engine or the second processing engine based on a type of the query received from the plurality of client computing devices by the common agent layer.

14. The method of claim 10, further comprising: generating a physical log and a logical log based on the data; replicating the physical log or the logical log for building data pages; and organizing data in the data pages in the first format for use by the first processing engine.

15. The method of claim 14, further comprising: replicating the logical log and distributing to the second component of the storage device; and categorizing copies of the logical log into a plurality of partitions based on a predetermined partitioning scheme.

16. The method of claim 15, further comprising: applying at least one portion of the copy of the logical log in order to an incremental data store of a corresponding partition of the plurality of partitions, wherein data in the incremental data store is stored in a row format and ordered based on a log sequence number (LSN) associated with the at least one portion of the copy of the logical log; and flushing data from the delta data store to a base data store based on one or more predetermined rules, wherein data in the base data store is organized in a columnar format for use by the second processing engine.

17. A non-transitory computer-readable storage medium comprising computer-readable instructions that, when executed by a system, cause the system to perform operations comprising: capturing data in real-time by a first processing engine, the first processing engine configured to perform online transaction processing; organizing the data in a first format in a first component of a storage device for use by the first processing engine; propagating the data to a second component of the storage device; organizing the data in a second format in the second component of the storage device for use by a second processing engine, the second processing engine configured to perform online analytical processing, wherein the first processing engine, the second processing engine, and the storage device are modularized and configured to be decoupled from one another; and extracting metadata from the data by a centralized metadata service; generating a metadata version by the centralized metadata service based on the extracted metadata; and aligning the metadata version with a sequence of data manipulation language by the centralized metadata service.

18. The non-transitory computer-readable storage medium of claim 17, the operations further comprising: dispatching, by a common agent layer, a query to the first processing engine or the second processing engine based on a type of the query received from a plurality of computing devices.

Images