Coordinating transactions across databases and open format tables

Abstract

In example embodiments, a solution for coordinating transactions across both open format tables on relational databases and object stores is provided. This technology addresses the challenge of integrating open table formats such as Delta and Iceberg with relational databases like SAP HANA. These technical challenges include subtleties of database transaction isolation, asynchronous parallelism, and handling instability. More specifically, transactions are broken down into smaller, independent sub-transactions. Each sub-transaction is executed independently. If a sub-transaction fails, a compensating action is performed to undo its effects. This approach ensures that the entire transaction is applied atomically.

Description

Technical Field

[0001] This disclosure relates to database management systems, and more specifically to coordinating transactions across relational databases and object storage devices using open format tables to ensure atomicity and consistency. Background Technology

[0002] In recent years, the integration of open table formats (such as Delta and Iceberg) with relational databases has become increasingly important for managing large datasets in lakehouse architectures. A lakehouse architecture is a modern data management approach that combines the advantages of data lakes and data warehouses, providing a unified platform for both analytical and transactional workloads. It addresses the limitations of traditional data lakes and data warehouses by providing a single storage tier that manages all types of data, including structured, semi-structured, and unstructured data, typically using scalable and cost-effective object storage systems. Lakehouses support Atomicity, Consistency, Isolation, and Durability (ACID) transactions, ensuring data integrity and consistency across operations, allowing for reliable data updates and concurrent access by multiple users or applications. Summary of the Invention

[0003] Embodiments of this disclosure relate to a system. The system includes: at least one hardware processor; a non-transitory computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations, the operations including: receiving a database transaction at a database management system; splitting the database transaction into a plurality of independent sub-transactions, the plurality of independent sub-transactions including a first sub-transaction corresponding to a first object in the database management system and a second sub-transaction corresponding to a second object in a remote open tabular storage system separate from and different from the database management system; recording the second sub-transaction in a meta-store of the database management system; sending the second sub-transaction to the remote open tabular storage system for execution and updating of the second object; updating the first object in the database management system by executing the first sub-transaction; based on determining that the update of the first object or the update of the second object has failed, executing a plurality of compensation transactions to restore the first object and the second object to an earlier state, the compensation transactions for restoring the second object being based on data in the meta-store; and clearing all data in the meta-store. Attached Figure Description

[0004] Figure 1 This is a block diagram illustrating a system according to an example embodiment.

[0005] Figure 2This is a flowchart illustrating a method for performing a registration and deregistration process according to an example embodiment.

[0006] Figure 3 This is a flowchart illustrating a method for performing a dirtiness check according to an example embodiment.

[0007] Figure 4 This is a diagram illustrating a parallelized workflow according to an example embodiment.

[0008] Figure 5 This is a diagram illustrating a method for managing transactions according to an example embodiment.

[0009] Figure 6 This is a block diagram illustrating the software architecture according to an example embodiment.

[0010] Figure 7 A graphical representation of a machine in the form of a computer system is shown, wherein a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein. Detailed Implementation

[0011] In view of the above-described embodiments of the subject matter, this application discloses the following list of examples, wherein a feature of a single example or multiple features of an example, combinations thereof, and optionally combinations thereof with one or more features of one or more other examples are also examples falling within the disclosure of this application.

[0012] Lakeware architecture includes mechanisms for schema implementation and evolution, enabling users to define and modify data schemas as needed while maintaining data quality and consistency. It supports business intelligence and machine learning workloads, allowing organizations to perform a wide range of data analytics tasks on a single platform. Lakeware architecture decouples storage devices from compute resources, allowing for independent scaling of each component, which provides flexibility in resource allocation and cost management.

[0013] Lakewarehousing typically utilizes open data formats such as Apache Parquet, Delta Lake, or Apache Iceberg to ensure compatibility with various data processing engines and tools. They also support real-time data extraction and processing, enabling timely insights and decision-making. By integrating the capabilities of data lakes and data warehouses, lakewarehousing architectures provide a comprehensive solution for managing large volumes of diverse data, supporting both batch and real-time processing, and enabling advanced analytics and machine learning applications.

[0014] Open Tabular Format (OPF) is a data storage format designed to facilitate efficient data management and processing in distributed environments, particularly in data lakes and lake warehouse architectures. These formats are characterized by their ability to support multi-engine read and write operations, allowing different data processing engines to access and manipulate data simultaneously. OPF is commonly used to store large datasets on object storage systems, providing a flexible and scalable solution for data analysis.

[0015] Open Table Format allows for multi-engine read and write operations on tables stored in object storage, offering significant advantages for analytical purposes. However, integrating these tables with relational databases such as SAP HANA™ from Walldorf SAP SE in Germany presents challenges, particularly in maintaining the atomicity and consistency of transactions across native relational tables and remote open format tables. HANA is an in-memory database. An in-memory database (also known as an in-memory database management system) is a type of database management system that relies primarily on main memory for storing computer data. This contrasts with database management systems that use disk storage mechanisms. In-memory databases are traditionally faster than disk-based databases because disk access is slower than memory access.

[0016] Existing solutions often lack the ability to effectively coordinate transactions across these disparate systems. Open Format Tables typically do not support two-phase commit protocols or external coordinators, resulting in operations being committed as a single atomic transaction without coordination. This limitation poses a significant challenge for applications requiring a degree of atomicity and consistency in cloud deployments involving both relational databases and remote Open Format Tables. Therefore, there is an urgent need for a solution that can provide atomic transactions across these systems, ensuring reliable and consistent data management in complex deployment scenarios.

[0017] More specifically, it is desirable to be able to deploy collections of artifacts where some objects are HANA objects and some are delta tables. It is also desirable to ensure atomicity in such deployments, making it possible to roll back the entire deployment, to roll back to previous transaction savepoints within the deployment, and to maintain eventual consistency of the collection of objects. This relieves developers and end users of concerns about the atomicity of orchestrating individual deployments. Additionally, it is desirable to support collections of Data Definition Language (DDL) transactions, particularly DDL transactions that create, discard, or modify tables, and those that default columns to null.

[0018] However, there are technical challenges in achieving these goals. Each incremental table has a transaction domain independent of other incremental tables. Furthermore, it's not possible to lock incremental tables in the same way as locking HANA tables. Additionally, the data lake system catalog hosting the incremental shared data is another separate transaction domain. Moreover, there is no central transaction manager for the collection of incremental tables. Finally, incremental transactions only have two isolation levels (serializable and write-serializable). Therefore, exhibiting atomicity properties in deployments that include both incremental tables and HANA objects is difficult.

[0019] In an example embodiment, a solution is provided for coordinating transactions across both relational databases and Open Format tables on object storage devices. This technique addresses the challenges of integrating Open Table Format tables such as Delta and Iceberg with relational databases such as SAPHANA. More specifically, transactions are broken down into smaller, independent sub-transactions. Each sub-transaction executes independently. If a sub-transaction fails, a compensating action is performed to undo its effects. This approach ensures that the entire transaction is applied atomically.

[0020] The provided solution offers the ability to manipulate data in an open file format in a remote object store through a database in the same way that data in a database is manipulated. It provides a seamless transaction processing experience, encompassing both regular database objects and objects residing remotely in object file format. It binds different transaction domains together, although each transaction is committed independently. Therefore, multiple distinct transaction domains, including relational databases and one or more remote object stores in one or more data lake systems, are bound together and appear as a single transaction domain without the actual effort required to make them a single transaction domain.

[0021] The Saga pattern is a design pattern used in distributed systems, particularly microservice architectures, to manage long-running transactions across multiple services by dividing them into a series of local transactions, each with a compensating transaction for rollback in case of failure. If a transaction within a Saga fails, the pattern ensures that a compensating transaction is executed to revert the effects of previously completed transactions. This approach allows the system to gracefully handle failures and maintain consistency. However, the Saga pattern has several drawbacks. First, the Saga pattern relies on an external (centralized) orchestrator. Second, the Saga pattern is designed to manage transactions across different data lake systems but does not include the ability to manage transactions between local databases and remote data lake systems simultaneously.

[0022] In the example embodiment, a novel pattern is established that manages long-running transactions across multiple services by dividing them into a series of local transactions, each with a compensating transaction for rollback in case of failure. This is done without using an external orchestrator and also in a manner that manages transactions between the local database and the remote data lake system. The compensation logic is also table-specific. Atomicity is ensured by writing transactions within the local relational database separately to the metastore, providing additional protection against failures.

[0023] DDL (Data Definition Language) is a set of Structured Query Language (SQL) statements used to modify the metadata of a database. Common DDL commands include CREATE (for creating database objects), ALTER (for modifying existing database objects), DROP (for deleting existing database objects), TRUNCATE (for removing records from a table while preserving its structure), COMMENT (for adding comments to database objects), and RENAME (for renaming database objects). DDL commands change the structure of the database, not the data itself.

[0024] In the example implementation, the solution described above provides DDL operations that can be safely executed on multiple incremental tables, as well as the creation or updating of virtual tables pointing to incremental tables. DDL operations can also be executed on native HANA tables / objects. This solution overcomes several technical challenges in supporting atomic behavior for such deployment scenarios. First, the incremental log protocol only supports single-table transactions. If an application needs to perform updates across multiple incremental tables, the atomicity of such updates cannot be guaranteed. Second, the transaction domains of incremental tables and the Hana system are completely different. When a logical transaction spans a set of incremental tables and HANA artifacts, incremental transactions already committed on individual tables cannot be easily rolled back if they must be rolled back. Third, incremental tables cannot be locked during deployment. Therefore, although deployment operations correspond to a set of incremental tables, each of these tables can be modified by other operations that are not part of the deployment. For example, when an incremental table undergoes an ALTER operation (such as when adding a column) as part of a deployment within a single table transaction, the table can be modified by data update operations from another job.

[0025] A meta-transaction can be defined as a collection of independent transactions initiated by an application, each corresponding to a modification of an incremental table, where each transaction operates within its own transaction domain. From the application's perspective, such independent transactions form a single logical transaction. In other words, a meta-transaction is a logical transaction that spans independent transactions.

[0026] For example, a meta-transaction can span a set of transactions executed to apply DDL to an incremental set of tables. For ease of discussion, a transaction that is part of a meta-transaction can be referred to as a sub-transaction.

[0027] In the example embodiment, the atomicity requirement for lifecycle management of the incremental table is provided by an orchestrator component for the meta-transaction, which ensures that compensating transactions are executed if the meta-transaction needs to be aborted for some reason. These compensating transactions may include, for example, actions such as rolling back sub-transactions to restore committed sub-transactions that were part of the meta-transaction.

[0028] Because each subtransaction is committed independently of the others, the entire metatransaction can be in a state where one of the subtransactions can be considered committed, but another subtransaction can be in progress, aborted, or rolled back. However, this will be a transient state until the orchestrator executes a compensating transaction to eventually bring the system back to the state where the metatransaction was atomically applied. This atomic behavior provided by such an orchestration mechanism can be referred to as the "final atomicity" of the metatransaction.

[0029] In the context of lifecycle management, a meta-transaction encapsulates a set of transactions on the corresponding set of incremental tables in the deployment. HANA transactions are also required when HANA virtual tables are mapped to these incremental tables or when other HANA artifacts are involved during deployment. In this case, the HANA transaction is also part of the meta-transaction that updates the incremental tables and HANA artifacts. Unlike HANA transactions, meta-transactions only provide a final atomicity guarantee.

[0030] The previously discussed final atomic orchestrator component uses information about the set of actions as part of a sub-transaction because it needs to perform compensating transactions as needed. For this purpose, in the example embodiment, the actions of all ongoing meta-transactions can be stored in a layer called the meta-store. The information in the meta-store is only used to perform compensating actions (i.e., undo actions) on the incremental table.

[0031] For example, the metastore could be a HANA table in a HANA system, where deployed applications can connect to and orchestrate deployments across HANA and incremental tables. The metastore maintains the state necessary for the final atomic orchestrator component to perform such logical undo operations; this includes a list of DDL operations performed on incremental tables and the version of the incremental table on which the DDL will be applied. For example, it might contain older / newer snapshot versions, DDLs, etc.

[0032] Meta storage is populated as part of DDL processing.

[0033] Figure 1This is a block diagram illustrating system 100 according to an example embodiment. Deployment application 102 deploys lifecycle management artifacts (such as incremental tables, virtual tables, and HANA artifacts) to database management system 104. In the example embodiment, database management system 104 may be a HANA service. Here, for example, deployment application 102 wants to perform a single operation to store data. At database management system 104, this single operation is a meta-transaction consisting of two independent sub-transactions. The first sub-transaction creates virtual table 108 within database management system 104. The second sub-transaction creates incremental share 110 in data lake system 112. More specifically, data lake adapter 114 may provide a SQL application interface (API) to perform DDL on incremental tables (such as incremental table 116). More specifically, new SQL DDL syntax may be supported, which utilizes a remote adapter to perform incremental DDL execution on, for example, a Spark service. Here, incremental table 116 and virtual table 108 can be created in a single DDL with the new supported syntax and data lake adapter 114, which is capable of creating virtual files for read access and also remotely requesting DDL execution. As previously mentioned, metastore 120 is used to store transactions written to database management system 104 for future rollback (if needed).

[0034] The lifecycle management module 122 performs the above operations. It contains the final autonomous component 124, which executes and / or guides sub-transactions and writes information about the appropriate sub-transactions to the meta-store 120. The compensation component 126 then uses the information in the meta-store 120 to perform one or more compensation operations in case of any issues.

[0035] Even if the operation is rolled back, the parameters used for compensation need to be available. Therefore, registration is performed and committed in a separate internal transaction (which may be called a self-contained transaction), rather than within a user transaction that may eventually be rolled back. In contrast to registration, deregistration needs to be performed atomically along with the user operation (therefore, deregistration entries are only retrieved from the metastore if and only if the user operation is committed). For this purpose, deregistration is performed using the user transaction itself.

[0036] As a side effect, in some cases, registrations are not visible to user transactions. This can be understood based on the isolation characteristics of database transactions. Because autonomous transactions begin after user transactions and commit independently, for certain (more stringent) isolation level settings, registration changes performed by them are not visible to user transactions. This can be a problem for the deregistration process, since deregistration discards registered entries, which must be visible. To overcome this problem, in the example embodiment, the metastore 120 is constructed as a log, and any state change of a registered entry inserts a new record into the log, rather than modifying the entry in place. Finally, a background job clears entries marked as pending deregistration.

[0037] Figure 2 This is a flowchart illustrating a method 200 for performing a registration and deregistration process according to an example embodiment. At operation 202, a DDL transaction is received. At operation 204, a lakeware commit process is triggered. At operation 206, an autonomous transaction is triggered. At operation 208, DDL registration is performed. At operation 210, the DDL is committed.

[0038] At operation 212, the DDL is cancelled. At operation 214, the remote sub-transaction is executed. At operation 216, the local sub-transaction is executed. At operation 218, the meta-transaction is committed (or rolled back). It is worth noting that operations 212 and 218 are executed within the same transaction context.

[0039] Because the compensation meta-operation (executed during user operation rollback) is triggered asynchronously, the database management system and data lake system may be temporarily out of sync. During this period, further user operations on remote objects can be prevented by checking the state of the remote object from the meta-store. However, the remote object can be operated on by some other user transaction; in this case, the object's registration entry should exist in the meta-store. Due to transaction isolation features, this registration may not be visible to the current user transaction, and therefore the check should be performed via an autonomous transaction.

[0040] If a remote object has been modified and subsequently left in an error state (pending compensation action) by some previous operation within the current user transaction itself, then the entry will be invisible outside the current transaction, especially for autonomous transactions. Therefore, similar checks need to be performed within the current transaction as well.

[0041] Figure 3This is a flowchart illustrating a method 300 for performing a contamination check according to an example embodiment. At operation 302, a DDL transaction is received. At operation 304, a discretionary transaction is triggered. At operation 306, it is determined whether the discretionary transaction has been contaminated by another transaction. If so, the transaction is aborted at operation 308. If not, it is determined at operation 310 whether the discretionary transaction has been contaminated by a DDL transaction. If so, it is determined at operation 312 whether the last operation failed. If so, the transaction is aborted at operation 308. If not, or if it is determined at operation 310 that the discretionary transaction has not been contaminated by a DDL transaction, the DDL operation and registration can begin at operation 314. It is noteworthy that operations 312 and 314 are performed within a DDL transaction context.

[0042] In some example implementations, a memory queue is used, such that whenever a DDL is rolled back, a rollback is triggered, inserting a unique identifier into the queue. A background job is then periodically invoked. This background job performs three operations. First, if this is the first invocation since server startup, crash recovery is performed by assuming that all registry entries in the metastore that are not in a pending post-commit or pending deregistration state should correspond to DDLs that could not be successfully committed before the crash, and therefore these are rolled back in serial order. Second, the memory queue is emptied, and compensation is performed in serial order. Third, the metastore is checked for all pending post-commit or deregistration entries, and the corresponding operations are performed serially.

[0043] However, such an implementation presents several technical challenges. Specifically, compensation on individual objects does not need to be serialized. Furthermore, the intervals between periodic jobs introduce near-deterministic latency after compensation / commit. These issues become particularly critical due to the inherently slow nature of the connection to the SPARK terminal and its performance characteristics. Serialization further exacerbates this. From the user's perspective, objects remain unavailable for writing until compensation is performed.

[0044] Thus, a parallelized workflow is introduced in the example embodiment. Here, rollback is used after a commit. Furthermore, a contextual asynchronous workflow is introduced. Each post-compensation or commit rollback establishes a queue and asynchronously triggers a job to execute the operation. Any new compensations or commits arriving while the job is being executed simply enqueue the request. The asynchronous job uses the requests in the queue. Once empty, it discards the queue and terminates itself. If a job fails to trigger (e.g., due to a temporary resource crash), the job is added to a pending queue. Each object has an independent workflow, thus implicitly ensuring full parallelism.

[0045] Then, a daemon is set up to ensure that all jobs in the pending queue are eventually invoked. As mentioned earlier, during the first run after a restart, the asynchronous execution memory state is bootstrap from the metastore. Hidden configuration parameters can be introduced, such as the maximum number of concurrent asynchronous jobs and the maximum wait time for threads fetching asynchronous jobs.

[0046] Figure 4 This is a diagram illustrating a parallelized workflow 400 according to an example embodiment. At operation 402, the DLL operation begins. At operation 404, DDL registration occurs. This places information about the DDL into the meta-store 406. At operation 408, remote and local operations are executed. Once these are complete, either rollback 410 or commit 412 is performed. In either case, a daemon 414 is involved to ensure all jobs in the pending call queue 416 are accounted for. This is performed asynchronously. Then DDL deregistration 418 occurs.

[0047] The asynchronous operations following compensation and commit are system-initiated, but like all other external spark operations, they are susceptible to various failures (network issues, etc.). Such failures can leave remote objects in an inconsistent state, and further user operations from the HANA database can be prevented from accessing them. Therefore, having the necessary retry mechanisms in appropriate locations is useful.

[0048] The range of factors that can cause a Spark program to fail is wide. While many of these error conditions are transient and can be resolved with quick retries, others may require configuration changes or be caused by persistent interruptions. This necessitates a multi-scale retry mechanism that includes both short-term and long-term retries. Asynchronous execution frameworks provide such functionality.

[0049] Self-healing can be achieved by retrying the operation a set number of times every set number of seconds within an asynchronous call, with the thread occupied for the entire duration of a short-term retry. In the long term, the asynchronous job can be called again at intervals starting from a certain offset in seconds, increasing the duration exponentially until a maximum interval is reached, until a maximum retry duration in seconds has elapsed.

[0050] Regarding the compensation component, user actions can include multiple steps, and the precise point of failure can occur anywhere within these steps, thus opening up the possibility of multi-level error states. Therefore, the compensation component is designed to correctly handle all possibilities. The compensation action itself can also include multiple steps, and therefore the compensation itself can fail at any point. Furthermore, compensation can be attempted an arbitrary number of times, and each attempt can fail arbitrarily. In all these cases, subsequent attempts should attempt to perform the necessary processing.

[0051] The Delta Lake protocol includes very limited support for atomic operations or object locking to prevent concurrent writes from multiple applications. The only meaningful tools available are the ability to add some identification information to the table's incremental log (Deltalog) as part of a transactional write to it, and the ability to index the incremental log by table version.

[0052] In an example implementation, support can be introduced to provide consistency in such a way that for each Delta Lake DDL run from HANA, a unique GUID is generated, which is inserted into the incremental log after the DDL execution. The compensator then uses this GUID as "proof" of consistency, based on a deterministic sequence of steps that, according to the DDL execution itself and the GUID insertion into the log, should create a predetermined number of versions. Therefore, depending on the position of the GUID in the incremental log, the compensator logic can backtrack to check for any irrelevant versions that may have been potentially created by concurrent user operations, and report any errors in this case (to prevent data loss).

[0053] While the above mechanism can identify concurrent user operations during the execution of the DDL itself, ensuring that no user operations occur between the rollback of a DDL and its compensation is much more difficult. In principle, this strict consistency guarantee can be implemented as pushdown automata-style logic, which models the incremental log as a stack and each DDL and its compensation as a push / pop operation (mirroring the semantics of nested-savepoint rollbacks). However, physically, the incremental log is a monotonically increasing sequence of records, but this can be considered. The scheme can be described in pseudocode as follows:

[0054]

[0055]

[0056]

[0057]

[0058] Figure 5 This is a diagram illustrating a method 500 for managing transactions according to an example embodiment. Here, operations with solid-line boundaries are performed by an external API used by the deployed application, while operations with dashed-line boundaries are performed by internal logic in a lifecycle management component built into the database management system.

[0059] At operation 502, the deployment application initiates a meta-transaction. This also starts a HANA DDL transaction on the database management system side. At operation 504, the incremental table DDL is executed. At operation 506, information about the incremental table DDL is recorded in the meta-store.

[0060] At operation 508, updates are performed on the HANA virtual tables to synchronize them with the incremental tables updated in operation 504. At operation 510, additional HANA DDL operations are performed. If no errors occur during operations 504, 508, or 510, at operation 512, the meta-transaction is committed in the meta-store, which also commits the HANA DDL transactions. At operation 514, meta-transaction information is cleared from the meta-store.

[0061] If an error occurs during one or more of operations 504, 508, or 510, the entire meta-transaction is rolled back at operation 516. At operation 518, the meta-store is used to restore all affected incremental tables to their previous versions. At operation 520, the HANA transaction is rolled back. At operation 522, the meta-transaction information is cleared from the meta-store.

[0062] In view of the above-described embodiments of the subject matter, this application discloses the following list of examples, wherein a feature of a single example or multiple features of an example, combinations thereof, and optionally combinations thereof with one or more features of one or more other examples are also examples falling within the disclosure of this application.

[0063] Example 1 is a system comprising: at least one hardware processor; and a non-transitory computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations including: receiving a database transaction at a database management system; splitting the database transaction into multiple independent sub-transactions, the multiple independent sub-transactions including a first sub-transaction corresponding to an object in the database management system and a second sub-transaction corresponding to an object in a remote open tabular storage system separate from and different from the database management system; recording the second sub-transaction in the meta-store of the database management system; sending the second sub-transaction to the remote open tabular storage system for execution and updating of the second object; updating the first object in the database management system by executing the first sub-transaction; based on determining that the update of the first object or the update of the second object has failed, executing multiple compensation transactions to restore the first object and the second object to an earlier state, the compensation transactions for restoring the second object being based on data in the meta-store; and clearing all data in the meta-store.

[0064] In Example 2, the subject of Example 1 includes, wherein multiple independent sub-transactions further include a third sub-transaction, the third sub-transaction corresponding to an object in a second remote open table format storage system that is separate from and different from the database management system and the remote open table format storage system.

[0065] In Example 3, the topics of Examples 1-2 include, where the database management system is an in-memory database management system.

[0066] In Example 4, the subject of Examples 1-3 is included, where the first and second sub-transactions are both DDL transactions.

[0067] In Example 5, the subject of Examples 1-4 includes the following: the first sub-transaction is used to update a virtual table in the database management system when it is executed.

[0068] In Example 6, the subject of Examples 1-5 includes, wherein the operation further includes: executing one or more additional sub-transactions on a database management system; and wherein determining includes determining that the update of a first object, the update of a second object, or the execution of one or more additional sub-transactions has failed.

[0069] In Example 7, the subject of Examples 1-6 includes: where the first sub-transaction and the second sub-transaction are executed asynchronously.

[0070] Example 8 is a method comprising: receiving a database transaction at a database management system; splitting the database transaction into multiple independent sub-transactions, the multiple independent sub-transactions including a first sub-transaction corresponding to an object in the database management system and a second sub-transaction corresponding to an object in a remote Open Table Format Storage system separate from and different from the database management system; recording the second sub-transaction in the meta-store of the database management system; sending the second sub-transaction to the remote Open Table Format Storage system for execution and updating of the second object; updating the first object in the database management system by executing the first sub-transaction; based on determining that the update of the first object or the update of the second object has failed, executing multiple compensation transactions to restore the first object and the second object to an earlier state, the compensation transactions for restoring the second object being based on data in the meta-store; and clearing all data in the meta-store.

[0071] In Example 9, the subject of Example 8 includes, wherein multiple independent sub-transactions further include a third sub-transaction, the third sub-transaction corresponding to an object in a second remote open table format storage system that is separate from and different from the database management system and the remote open table format storage system.

[0072] In Example 10, the topics of Examples 8-9 include, where the database management system is an in-memory database management system.

[0073] In Example 11, the subject of Examples 8-10 includes the fact that both the first and second sub-transactions are DDL transactions.

[0074] In Example 12, the subject of Examples 8-11 includes the following: a first sub-transaction is executed to update a virtual table in a database management system.

[0075] In Example 13, the subject of Examples 8-12 includes executing one or more additional sub-transactions on a database management system; and wherein it is determined that the update of a first object, the update of a second object, or the execution of one or more additional sub-transactions has failed.

[0076] In Example 14, the subject of Examples 8-13 includes: where the first sub-transaction and the second sub-transaction are executed asynchronously.

[0077] Example 15 is a non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause one or more processors to perform operations including: receiving a database transaction at a database management system; splitting the database transaction into multiple independent sub-transactions, the multiple independent sub-transactions including a first sub-transaction corresponding to an object in the database management system and a second sub-transaction corresponding to an object in a remote Open Table Format Storage System separate from and different from the database management system; recording the second sub-transaction in the meta-store of the database management system; sending the second sub-transaction to the remote Open Table Format Storage System for the execution and updating of the second object; updating the first object in the database management system by executing the first sub-transaction; based on determining that the update of the first object or the update of the second object has failed, executing multiple compensation transactions to restore the first object and the second object to an earlier state, the compensation transactions for restoring the second object being based on data in the meta-store; and clearing all data in the meta-store.

[0078] In Example 16, the subject of Example 15 includes a third sub-transaction that corresponds to an object in a second remote open table format storage system that is separate from and different from the database management system and the remote open table format storage system.

[0079] In Example 17, the topics of Examples 15-16 include, where the database management system is an in-memory database management system.

[0080] In Example 18, the subject of Examples 15-17 includes the fact that both the first and second sub-transactions are DDL transactions.

[0081] In Example 19, the subject of Examples 15-18 includes the following: a first subtransaction is executed to update a virtual table in a database management system.

[0082] In Example 20, the subject of Examples 15-19 includes, wherein the operation further includes: executing one or more additional sub-transactions on a database management system; and wherein determining includes determining that the update of a first object, the update of a second object, or the execution of one or more additional sub-transactions has failed.

[0083] Example 21 is at least one machine-readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement any one of Examples 1-20.

[0084] Example 22 is an apparatus that includes components for implementing any one of Examples 1-20.

[0085] Example 23 is a system for implementing any one of Examples 1-20.

[0086] Example 24 is a method for implementing any one of Examples 1-20.

[0087] Figure 6 This is a block diagram 600 illustrating software architecture 602, which can be installed on any one or more of the aforementioned devices. Figure 6 This is merely a non-limiting example of a software architecture, and it should be understood that many other architectures can be implemented to facilitate the functionality described herein. In various examples, software architecture 602 consists of, for instance,... Figure 7 The hardware implementation of machine 700 includes a processor 710, memory 730, and input / output (I / O) components 750. In this example architecture, software architecture 602 can be conceptualized as a stack of layers, where each layer provides specific functionality. For example, software architecture 602 includes layers such as operating system 604, libraries 606, frameworks 608, and application 610. Operationally, consistent with some examples, application 610 invokes API call 612 through the software stack and receives message 614 in response to API call 612.

[0088] In various implementations, operating system 604 manages hardware resources and provides common services. Operating system 604 includes, for example, a kernel 620, services 622, and drivers 624. Consistent with some examples, kernel 620 serves as an abstraction layer between hardware and other software layers. For example, kernel 620 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Service 622 can provide other common services for other software layers. According to some examples, driver 624 is responsible for controlling or interfacing with the underlying hardware. For example, driver 624 may include a display driver, camera driver, BLUETOOTH® or BLUETOOTH® Low Energy driver, flash memory driver, serial communication driver (e.g., Universal Serial Bus (USB) driver), Wi-Fi® driver, audio driver, power management driver, etc.

[0089] In some examples, library 606 provides low-level public infrastructure utilized by application 610. Library 606 may include system libraries 630 (e.g., the C standard library) that provide functions such as memory allocation, string manipulation, and mathematical functions. Furthermore, library 606 may include API libraries 632, such as media libraries (e.g., libraries for supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group-4 [MPEG4], Advanced Video Coding [H.264 or AVC], Moving Picture Experts Group Layer-3 [MP3], Advanced Audio Coding [AAC], Adaptive Multi-Rate [AMR] audio codecs, Joint Picture Experts Group [JPEG or JPG], or Portable Web Graphics [PNG]), graphics libraries (e.g., an OpenGL framework for rendering in two-dimensional [2D] and three-dimensional [3D] in a graphics context on a display), database libraries (e.g., SQLite for providing various relational database functions), web libraries (e.g., WebKit for providing web browsing functionality), and so on. Library 606 may also include a wide variety of other libraries 634 to provide many other APIs to application 610.

[0090] Based on some examples, framework 608 provides advanced public infrastructure that can be utilized by application 610. For example, framework 608 provides various GUI functions, advanced resource management, advanced location services, etc. Framework 608 can provide a wide range of other APIs that can be utilized by application 610, some of which may be specific to a particular operating system 604 or platform.

[0091] In the examples, application 610 includes a home application 650, a contacts application 652, a browser application 654, a book reader application 656, a location application 658, a media application 660, a messaging application 662, a game application 664, and a variety of other applications, such as third-party applications 666. According to some examples, application 610 is a program that performs functions defined in the program. Various programming languages ​​can be used to create one or more of applications 610 constructed in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, third-party application 666 (e.g., an application developed by an entity other than a platform vendor using the Android™ or iOS™ Software Development Kit [SDK]) can be mobile software that runs on a mobile operating system (such as iOS™, Android™, Windows® Phone, or another mobile operating system). In this example, third-party application 666 may invoke API calls 612 provided by operating system 604 to facilitate the functions described herein.

[0092] Figure 7 A graphical representation of a machine 700 in the form of an example computer system is shown, wherein a set of instructions can be executed to cause the machine 700 to perform any or more methods discussed herein. Specifically, Figure 7 A graphical representation of a machine 700 in an example form of a computer system is shown, wherein instructions 716 (e.g., software, programs, applications, applets, apps, or other executable code) are executable to cause the machine 700 to perform any or more methods discussed herein. For example, instructions 716 may cause the machine 700 to perform... Figure 5 Method 500. Additionally or alternatively, instruction 716 can implement... Figures 1-5Instruction 716 transforms a general, unprogrammed machine 700 into a specific machine 700 programmed to perform the described and illustrated functions in the described manner. In alternative examples, machine 700 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 700 can operate as a server machine or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 700 can be, but is not limited to, server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, network devices, network routers, network switches, bridges, or any machine capable of sequentially or otherwise executing the actions specified in instruction 716 to be performed by machine 700. Furthermore, although only a single machine 700 is shown, the term "machine" should also be considered as including a collection of machines 700, which individually or jointly execute instructions 716 to perform any or more of the methods discussed herein.

[0093] Machine 700 may include processor 710, memory 730, and I / O components 750, which may be configured to communicate with each other, for example, via bus 702. In the example, processor 710 (e.g., a central processing unit [CPU], a reduced instruction set computing [RISC] processor, a complex instruction set computing [CISC] processor, a graphics processing unit [GPU], a digital signal processor [SP], an application-specific integrated circuit [ASIC], a radio frequency integrated circuit [RFIC], another processor, or any suitable combination thereof) may include processors 712 and 714, for example, capable of executing instructions 716. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions 716 simultaneously. Although... Figure 7 Multiple processors 710 are shown, but machine 700 may include a single processor 712 with a single core, a single processor 712 with multiple cores (e.g., a multi-core processor 712), multiple processors 712, 714 with a single core, multiple processors 712, 714 with multiple cores, or any combination thereof.

[0094] Memory 730 may include main memory 732, static memory 734, and memory cells 736, each of which may be accessed by processor 710 via bus 702. Main memory 732, static memory 734, and memory cells 736 store instructions 716 embodying any one or more of the methods or functions described herein. During execution of instructions 716 by machine 700, instructions 716 may also reside wholly or partially in main memory 732, static memory 734, memory cells 736, at least one of processor 710 (e.g., the processor's cache memory), or any suitable combination thereof.

[0095] I / O component 750 may include a wide variety of components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O component 750 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that I / O component 750 may include... Figure 7 Many other components are not shown. The grouping of I / O components 750 according to function is only for the purpose of simplifying the following discussion, and the grouping is by no means limiting. In various examples, I / O components 750 may include output components 752 and input components 754. Output components 752 may include visual components (e.g., displays, such as plasma display panels [PDP], light-emitting diode [LED] displays, liquid crystal displays [LCD], projectors, or cathode ray tubes [CRT]), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. Input components 754 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens that provide position and / or force for touch or touch gestures, or other haptic input components), audio input components (e.g., microphones), etc.

[0096] In another example, I / O component 750 may include biometric component 756, motion component 758, environmental component 760 or positioning component 762, and a variety of other components. For example, biometric component 756 may include components for detecting expressions (e.g., hand gestures, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), and recognizing a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 758 may include accelerometer components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Environmental component 760 may include, for example, a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting hazardous gas concentrations or measuring pollutants in the atmosphere for safety purposes), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 762 may include a position sensor component (e.g., a Global Positioning System [GPS] receiver component), an altitude sensor component (e.g., an altimeter or a barometer for detecting air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), etc.

[0097] Communication can be implemented using a wide variety of technologies. I / O component 750 may include communication component 764, operable to couple machine 700 to network 780 or device 770 via coupling 782 and coupling 772, respectively. For example, communication component 764 may include a network interface component or another suitable device interfaced with network 780. In further examples, communication component 764 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Bluetooth® communication components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components for providing communication via other modalities. Device 770 may be another machine or any peripheral device from a variety of peripheral devices (e.g., coupled via USB).

[0098] Furthermore, the communication component 764 may detect identifiers or include components operable to detect identifiers. For example, the communication component 764 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code [UPC] barcodes, multi-dimensional barcodes such as QR codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying audio signals from the tag). Additionally, various information can be derived via the communication component 764, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via detection of NFC beacon signals that can indicate a specific location, and so on.

[0099] Various memories (e.g., the memories of 730, 732, 734 and / or the memory of processor 710) and / or storage units 736 may store a set of one or more instructions 716 and data structures (e.g., software) that embody or are utilized by any one or more methods or functions described herein. These instructions (e.g., instructions 716), when executed by processor 710, cause various operations to implement the disclosed examples.

[0100] As used herein, the terms “machine storage medium,” “device storage medium,” and “computer storage medium” refer to the same thing and are used interchangeably. These terms refer to one or more storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions and / or data. Therefore, these terms should be considered to include, but are not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), field-programmable gate arrays (FPGAs), and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage medium,” “computer storage medium,” and “device storage medium” specifically exclude carrier waves, modulated data signals, and other such media, at least some of which are covered under the term “signal medium” discussed below.

[0101] In various examples, one or more portions of network 780 may be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a Common Old-Style Telephone Service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, network 780 or a portion of network 780 may include a wireless or cellular network, and coupling 782 may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or another type of cellular or wireless coupling. In this example, the coupling 782 can implement any of a variety of data transmission technologies, such as single-carrier radio transmission technology (1xRTT), evolved data optimization (EVDO) technology, general packet radio service (GPRS) technology, GSM evolution enhanced data rate (EDGE) technology, the 3rd generation partnership program (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high-speed packet access (HSPA), global microwave access interoperability (WiMAX), long-term evolution (LTE) standard, other standards defined by various standards-setting organizations, other remote protocols, or other data transmission technologies.

[0102] Instruction 716 can be sent or received over network 780 via a network interface device (e.g., a network interface component included in communication component 764) using a transmission medium and utilizing any of a plurality of known transmission protocols (e.g., HTTP). Similarly, instruction 716 can be sent or received to device 770 via a transmission medium via coupling 772 (e.g., peer-to-peer coupling). The terms “transmission medium” and “signal medium” refer to the same thing and can be used interchangeably in this disclosure. The terms “transmission medium” and “signal medium” should be considered to include any intangible medium capable of storing, encoding, or carrying instructions 716 for execution by machine 700, and include digital or analog communication signals or other intangible media to facilitate communication of such software. Therefore, the terms “transmission medium” and “signal medium” should be considered to include any form of modulated data signal, carrier wave, etc. The term “modulated data signal” means a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal.

[0103] The terms “machine-readable medium,” “computer-readable medium,” and “device-readable medium” refer to the same thing and may be used interchangeably in this disclosure. These terms are defined to include both machine storage media and transmission media. Therefore, these terms encompass both storage devices / media and carrier / modulated data signals.

Claims

1. A system for database management, comprising: At least one hardware processor; A non-transitory computer-readable medium stores instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations, the operations including: Receive database transactions at the database management system; The database transaction is split into multiple independent sub-transactions, the multiple independent sub-transactions including a first sub-transaction corresponding to a first object in the database management system and a second sub-transaction corresponding to a second object in a remote open table format storage system that is separate from and different from the database management system; The second sub-transaction is recorded in the metadata storage of the database management system; Send the second subtransaction to a remote Open Table Format storage system for the execution and updating of the second object; Update the first object in the database management system by executing the first sub-transaction; Based on the determination that the update of the first object or the update of the second object has failed, multiple compensation transactions are executed to restore the first and second objects to an earlier state. The compensation transaction for restoring the second object is based on data in the meta-store; and Clear all data in the metastore.

2. The system according to claim 1, wherein, The plurality of independent sub-transactions also include a third sub-transaction, which corresponds to an object in a second remote open tabular storage system that is separate from and different from the database management system and the remote open tabular storage system.

3. The system according to claim 1, wherein, The database management system is an in-memory database management system.

4. The system according to claim 1, wherein, Both the first and second sub-transactions are Data Definition Language (DDL) transactions.

5. The system according to claim 1, wherein, The first sub-transaction is used to update virtual tables in the database management system when it is executed.

6. The system according to claim 1, wherein, The operation also includes: Execute one or more additional sub-transactions on the database management system; and This determination includes determining that the update of the first object, the update of the second object, or the execution of any one or more additional sub-transactions has failed.

7. The system according to claim 1, wherein, The first and second sub-transactions are executed asynchronously.

8. A method for database management, comprising: Receive database transactions at the database management system; The database transaction is split into multiple independent sub-transactions, the multiple independent sub-transactions including a first sub-transaction corresponding to a first object in the database management system and a second sub-transaction corresponding to a second object in a remote open table format storage system that is separate from and different from the database management system; The second sub-transaction is recorded in the metadata storage of the database management system; Send the second subtransaction to a remote Open Table Format storage system for the execution and updating of the second object; Update the first object in the database management system by executing the first sub-transaction; Based on the determination that the update of the first object or the update of the second object has failed, multiple compensation transactions are executed to restore the first object and the second object to an earlier state. The compensation transaction for restoring the second object is based on the data in the meta storage. as well as Clear all data in the metastore.

9. The method according to claim 8, wherein, The plurality of independent sub-transactions also include a third sub-transaction, which corresponds to an object in a second remote open tabular storage system that is separate from and different from the database management system and the remote open tabular storage system.

10. The method according to claim 8, wherein, The database management system is an in-memory database management system.

11. The method according to claim 8, wherein, Both the first and second sub-transactions are Data Definition Language (DDL) transactions.

12. The method according to claim 8, wherein, The first sub-transaction is used to update virtual tables in the database management system when it is executed.

13. The method of claim 8, further comprising: Execute one or more additional sub-transactions on the database management system; as well as This determination includes determining that the update of the first object, the update of the second object, or the execution of any one or more additional sub-transactions has failed.

14. The method according to claim 8, wherein, The first and second sub-transactions are executed asynchronously.

15. A non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising: Receive database transactions at the database management system; The database transaction is split into multiple independent sub-transactions, the multiple independent sub-transactions including a first sub-transaction corresponding to a first object in the database management system and a second sub-transaction corresponding to a second object in a remote open table format storage system that is separate from and different from the database management system; The second sub-transaction is recorded in the metadata storage of the database management system; Send the second subtransaction to a remote Open Table Format storage system for the execution and updating of the second object; Update the first object in the database management system by executing the first sub-transaction; Based on the determination that the update of the first object or the update of the second object has failed, multiple compensation transactions are executed to restore the first object and the second object to an earlier state. The compensation transaction for restoring the second object is based on the data in the meta storage. as well as Clear all data in the metastore.

16. The non-transitory machine-readable medium according to claim 15, wherein, The plurality of independent sub-transactions also include a third sub-transaction, which corresponds to an object in a second remote open tabular storage system that is separate from and different from the database management system and the remote open tabular storage system.

17. The non-transitory machine-readable medium according to claim 15, wherein, The database management system is an in-memory database management system.

18. The non-transitory machine-readable medium according to claim 15, wherein, Both the first and second sub-transactions are Data Definition Language (DDL) transactions.

19. The non-transitory machine-readable medium according to claim 15, wherein, The first sub-transaction is used to update virtual tables in the database management system when it is executed.

20. The non-transitory machine-readable medium according to claim 15, wherein, The operation also includes: Execute one or more additional sub-transactions on the database management system; and This determination includes determining that the update of the first object, the update of the second object, or the execution of any one or more additional sub-transactions has failed.

Images