Multi-table and multi-query transactions in data lakehouse

Abstract

To provide multi-table and multi-query transactions in a data lakehouse. [Solution] A transaction targeting an object store employing the Open Table Format (OTF) can be detected by the database execution engine. The database execution engine then performs a decision step to determine which OTF extension level to apply to the detected transaction. Based on the decision step, the database execution engine then applies one or more OTF extension levels to the transaction. The database execution engine then ensures that the transaction is processed with the applied one or more OTF extension levels. The database execution engine then returns the result of the transaction processing to the computing device.

Claims

1. At least one processor, The system comprises at least one memory for storing instructions, and when the instructions are executed by the at least one processor, Detecting transactions targeting object stores that use the Open Table Format (OTF), Determining which of the multiple OTF extension levels to apply to the transaction, Based on the above decision, apply one or more OTF extension levels to the transaction, To ensure that the transaction is processed at one or more applied OTF extension levels, The result of processing the aforementioned transaction is returned to the computing device. A system that produces an action that includes [a certain action].

2. Applying the first OTF extension level is This involves securing a snapshot version, storing information about the transaction being processed, and using markers to prevent simultaneous operations from requesting the same snapshot version. Define the structure of a subblog that has redo and compensation steps for each transaction step. The system according to claim 1, including the following:

3. Applying the second OTF extension level This involves securing a snapshot version, storing information about the transaction being processed, and using markers to prevent simultaneous operations from requesting the same snapshot version. To operate a marker-based conflict and deadlock detection model between multiple tables in the aforementioned object store and The system according to claim 2, including the following:

4. The system according to claim 3, wherein applying a third OTF extension level includes implementing a global layer that manages multiple table snapshot versions to enable multi-version concurrent execution control globally.

5. The system according to claim 4, further comprising applying the third OTF extension level to implement a global version log summarizing the latest valid versions of the multiple tables.

6. The system according to claim 5, wherein each OTF extension level can be applied independently of the others of the plurality of OTF extension levels.

7. The system according to claim 6, further comprising applying the first OTF extension level to store the first marker in the metadata layer while the first transaction is being processed.

8. The system according to claim 7, wherein the operation further includes converting the first marker into a structure within the metadata layer during the commit of the first transaction.

9. The system according to claim 1, wherein the transaction targets multiple tables.

10. The system according to claim 1, wherein the transaction includes a plurality of queries.

11. Steps to detect transactions targeting object stores that use the Open Table Format (OTF), The steps include determining which of the multiple OTF extension levels to apply to the transaction, Based on the above decision, the steps include applying one or more OTF extension levels to the transaction, The steps include: ensuring that the transaction is processed at one or more applied OTF extension levels; The steps include: returning the result of processing the transaction to the computing device; Computer implementation methods, including those mentioned above.

12. The step of applying the first OTF extension level is, The steps include: securing a snapshot version, storing information about the transaction being processed, and using markers to prevent simultaneous operations from requesting the same snapshot version; A step of defining the structure of a subblog having redo and compensation steps for each transaction step. The computer implementation method according to claim 11, including the method described in claim 11.

13. The step of applying the second OTF extension level is, The steps include: securing a snapshot version, storing information about the transaction being processed, and using markers to prevent simultaneous operations from requesting the same snapshot version; The steps include: activating a marker-based contention and deadlock detection model between multiple tables in the object store; The computer implementation method according to claim 12, including the method described in claim 12.

14. The computer implementation method according to claim 13, wherein the step of applying a third OTF extension level includes the step of implementing a global layer that manages multiple table snapshot versions in order to enable multi-version concurrent execution control globally.

15. The computer implementation method according to claim 14, wherein the step of applying the third OTF extension level further includes the step of implementing a global version log that summarizes the latest valid versions of the plurality of tables.

16. The computer implementation method according to claim 15, wherein each OTF extension level can be applied independently of the others of the plurality of OTF extension levels.

17. The computer implementation method according to claim 16, wherein the step of applying the first OTF extension level further includes the step of storing the first marker in the metadata layer while the first transaction is being processed.

18. The computer implementation method according to claim 17, further comprising the step of converting the first marker in the metadata layer into a structure during the commit of the first transaction.

19. The computer implementation method according to claim 11, wherein the transaction targets multiple tables and the transaction includes multiple queries.

20. A non-temporary computer-readable storage medium storing instructions, wherein when the instructions are executed by at least one processor, Detecting transactions targeting object stores that use the Open Table Format (OTF), Determining which of the multiple OTF extension levels to apply to the transaction, Based on the above decision, apply one or more OTF extension levels to the transaction, To ensure that the transaction is processed at one or more applied OTF extension levels, The result of processing the aforementioned transaction is returned to the computing device. A non-temporary computer-readable storage medium that produces operations including [specific actions].

Images