Multi-table and multi-query transactions in data lake repositories

Abstract

The database execution engine can detect a transaction targeting object storage in an open table format (OTF). Next, the database execution engine performs a determination step to determine which OTF enhancement levels to apply to the detected transaction. Then, the database execution engine applies one or more OTF enhancement levels to the transaction based on the determination step. Next, the database execution engine causes the transaction to be processed with the applied one or more OTF enhancement levels. Then, the database execution engine returns results of the transaction processing to the computing device.

Description

Technical Field

[0001] This disclosure generally relates to processing database queries in a data lake. Background Technology

[0002] In recent years, a new type of data management system called lakehouse (LH) has emerged, combining the benefits of low-cost, open-format data lakes and transactional data warehouses on relational data. As data gravity drives data lakes, LHs for analytical query processing become crucial. LHs are used in a variety of applications, such as biomedical, deep learning, and analytics business applications. Contributing to this decision is LH's provision of cross-data system vendor interoperability through its open format, relatively low cost, high throughput, metadata information, and reliable data lake storage (e.g., Amazon Web Services (AWS) S3). LHs typically experience highly concurrent workloads, with 60% reads and 40% writes from independent applications and sources. Therefore, LHs are often characterized by elastic computing (e.g., Apache Spark, Presto, Trino) for processing data physically stored in data formats such as column-oriented Parquet or Optimized Row Columnar (ORC).

[0003] To leverage relational table semantics and data analytics capabilities for efficient and resilient query processing, Open Table Format (OTF) solutions such as Delta Lake Table, Apache Iceberg, or Apache Hudi manage the underlying data through a separate metadata layer on top of object storage. While OTF is designed to specify certain relational database management system (DBMS) functionalities, such as transactions for individual tables and rows, indexes, and time travel, it also has many inherent shortcomings in its metadata design, such as lack of support for multi-table transactions and recovery from aborts in concurrent, multi-query workloads. This can lead to non-repeatable reads, stale data, and high costs in production cloud systems. Summary of the Invention

[0004] In some implementations, a transaction is detected, where the transaction targets an object store using Open Table Format (OTF). Next, the database execution engine performs a determination step to determine which OTF enhancement levels to apply to the transaction. Then, based on the determination step, the database execution engine applies one or more OTF enhancement levels to the transaction. Next, the database execution engine causes the transaction to be processed using the applied one or more OTF enhancement levels. Finally, the database execution engine returns the result of the transaction processing to the computing device.

[0005] In the example, the first OTF enhancement level can utilize flags to preserve snapshot versions, store information about in-flight transactions, and prevent concurrent operations from declaring the same snapshot version. The first OTF enhancement level can also define a sub-log structure with redo and compensation steps for each transaction step. In this example, the second OTF enhancement level can activate a flag-based conflict and deadlock detection model across multiple tables in the target database. Furthermore, in this example, the third OTF enhancement level can implement a global layer for managing multiple table snapshot versions to globally enable multi-version concurrency control, and can also implement a global version log that aggregates the most recent valid versions of multiple tables. Each OTF enhancement level can be applied independently of the other OTF enhancement levels.

[0006] The document also describes a non-transitory computer program product (i.e., a physically embodied computer program product) that stores instructions that, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform the operations described herein. Similarly, a computer system is described that may include one or more data processors and memory coupled to the one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more of the operations described herein. Furthermore, the methods may be implemented by one or more data processors within a single computing system or distributed among two or more computing systems. Such computing systems may be interconnected and may exchange data and / or commands or other instructions via one or more connections (including connections via networks (e.g., the Internet, wireless wide area networks, local area networks, wide area networks, wired networks, etc.), via direct connections between one or more of the multiple computing systems, etc.

[0007] Details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the following description. Further features and advantages of the subject matter described herein will be apparent from the specification, drawings, and claims. Attached Figure Description

[0008] The accompanying drawings, which are incorporated in and form part of this specification, illustrate certain aspects of the subject matter disclosed herein and, together with the description, help to explain some principles associated with the disclosed embodiments. In the drawings,

[0009] Figure 1 A logic diagram of an example system based on some exemplary implementations of the current topic is shown;

[0010] Figure 2 A block diagram illustrating an example of a system based on some exemplary implementations of the current topic is shown;

[0011] Figure 3 Examples of data lake and reservoir architectures based on some exemplary implementations of the current topic are shown;

[0012] Figure 4 Examples of metadata layer designs for Iceberg tables and incremental lake tables are shown, based on some example implementations of the current topic;

[0013] Figure 5 Examples of tag integration based on some exemplary implementations of the current topic are shown;

[0014] Figure 6 Examples of physical and logical SAGA structures according to some exemplary implementations of the current topic are shown;

[0015] Figure 7 Examples of tag-based conflict and deadlock detection are shown, based on some exemplary implementations of the current topic;

[0016] Figure 8 Examples of global version logs are shown, based on some example implementations of the current topic;

[0017] Figure 9 The diagram illustrates an example of a system implemented according to some exemplary methods of the current topic;

[0018] Figure 10 The process for determining which levels to apply to a query targeting a data lake is illustrated according to some example implementations of the current topic;

[0019] Figure 11 The process for implementing a nested metadata layer in an object store is illustrated according to some example implementations of the current topic;

[0020] Figure 12A Examples of systems implemented according to some exemplary methods of the current topic are described; and

[0021] Figure 12B Another example of a system implemented according to some exemplary methods of the current topic is described. Detailed Implementation

[0022] A Data Lake Library (LH) is a data platform that combines various aspects of a data lake and a data warehouse into a single system. Data LH stores large amounts of raw data from various sources while preserving the original data format. Despite increasing research into LH storage system benchmarks, more efficient data formats, and LH query engines, many missing features of Open Table Format (OTF) have not been adequately investigated. Therefore, current LH designs impose constraints on the system to avoid missing OTF features. This specification focuses on three missing features: OTF does not support atomicity, consistency, isolation, and durability (ACID) multi-table and multi-query transactions, allowing only simple business logic (feature (i) complex transactions). Furthermore, due to OTF's table focus, OTF cannot globally guarantee isolation at all times, leading to inconsistencies and dirty reads under concurrency (feature (ii) isolation). Finally, OTF's conflict resolution is not suitable for concurrent changes, resulting in multiple aborts and higher transaction costs (feature (iii) recovery).

[0023] Existing OTFs and projects provide a critical foundation for analytics workloads, but OTFs can do much more with the techniques revealed in this paper. This document introduces LakeVilla, an extension that implements three missing features (i)-(iii) of existing OTF deployments at a composable level. LakeVilla's level is integrated into the original protocol, allowing each transaction the freedom to choose its desired guarantees. LakeVilla can be "constructively correct" by adapting the following techniques for various missing features: it can support complex transactions (i) by employing a tag-based conflict and deadlock detection model across all tables in the LH. Furthermore, to guarantee isolation (ii), a global version log can be implemented, which examines global relationships between tables and acts as a new global layer on object storage. Finally, for recovery (iii), logical snapshot version tags can be used with transaction sub-logs, as well as physical and logical undo and redo information, to define a greedy transaction model (similar to database records).

[0024] This specification identifies and analyzes the missing metadata-related features of LH and proposes a complementary solution to the current LH design called LakeVilla. The LakeVilla solution brings efficient recovery, multi-query and multi-table transactions, and transaction isolation to state-of-the-art OTFs (such as Apache Iceberg and Delta tables), with minimal impact on read performance and full compatibility with OTFs.

[0025] Now for reference Figure 1 The diagram depicts an example of a system 100 consistent with an implementation method consistent with the present subject. Figure 1As shown, system 100 may include cloud platform 130, which can provide resources that can be shared among multiple tenants. For example, cloud platform 130 may be configured to provide various services, including Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), etc., and these services can be accessed by one or more tenants of cloud platform 130. Figure 1 In the example, system 100 includes a first tenant 140A (labeled as a client) and a second tenant 140B (also labeled as a client), although system 100 may include any number of other tenants. For example, multitenancy enables multiple end-user devices (e.g., computers including applications) and multiple subscribers with their own end-user groups (which have their own client-specific isolation contexts) to access a given cloud service with shared resources via the Internet and / or other types of network 110 or (multiple) communication links.

[0026] Cloud platform 130 may include resources such as at least one computer (e.g., a server), data storage, and a network (including network devices) coupling the computer(s) and storage. Cloud platform 130 may also include other resources such as operating systems, hypervisors, and / or other resources to virtualize physical resources (e.g., via virtual machines) and provide deployment of applications (e.g., those offered on the cloud platform, and other resources) (e.g., via containers). Where cloud platform 130 includes and / or is coupled to a “public” cloud platform, services can be provided to clients or tenants on demand via the Internet. For example, resources at a public cloud platform may be operated and / or owned by a cloud service provider (e.g., Amazon Web Services, Azure), allowing physical resources at the cloud service provider to be shared by multiple tenants. Alternatively or additionally, cloud platform 130 may include and / or be coupled to one or more local servers, in which case some resources utilized by clients 140A-140B may be hosted on the entity’s own private servers (e.g., dedicated corporate servers operated and / or owned by the entity). Alternatively or additionally, cloud platform 130 may be considered a “hybrid” platform, which includes and / or is coupled to a combination of on-premises resources and resources hosted by a public or private cloud platform. For example, a hybrid platform may include a web server running in a public cloud, while application servers and / or databases are hosted on-premises (e.g., in a region controlled or operated by an entity such as a corporate entity).

[0027] In various embodiments, cloud platform 130 provides services to clients 140A-140B. Each service can be deployed via a container, which provides a package or bundle of software, libraries, and configuration data, enabling the cloud platform to deploy the service to one or more virtual machines, such as those providing services to client 140A, during runtime. Services may also include logic (e.g., instructions providing one or more steps of a process) and interfaces. Interfaces can be implemented as Open Data Protocol (OData) interfaces (e.g., HTTP messages can be used to create queries for resources identified by URIs), but interfaces can be implemented using other types of protocols, including those based on REST (Representational State Transition). Figure 1 In the example, an external REST type call can be used to send a query and receive a response from database 120.

[0028] Turn now Figure 2 A system diagram illustrating an example of a database system 200 is shown, based on one or more embodiments of the present subject. (Reference) Figure 2 Database system 200 may include one or more client devices 202, database execution engine 250, and one or more databases 290. For example... Figure 2 As shown, one or more client devices 202, a database execution engine 250, and one or more databases 290 can be communicatively coupled via a network 260. The one or more databases 290 can include various relational databases, including, for example, in-memory databases, column-based databases, row-based databases, etc. The database execution engine 250 can access database tables 295 stored in the one or more databases 290, where database tables 295 represent any number and type of tables.

[0029] In some example embodiments, one or more databases 290 may include relational databases. However, it should be understood that one or more databases 290 may include any type of database, including, for example, in-memory databases, hierarchical databases, object databases, object-relational databases, etc. For example, instead of including relational databases and / or in addition to including relational databases, one or more databases 290 may include graph databases, column stores, key-value stores, document stores, etc.

[0030] One or more client devices 202 may include processor-based devices, such as mobile devices, wearable devices, personal computers, workstations, Internet of Things (IoT) devices, etc. Network 260 may be a wired network and / or a wireless network, including, for example, a Public Land Mobile Network (PLMN), a Local Area Network (LAN), a Virtual Local Area Network (VLAN), a Wide Area Network (WAN), the Internet, etc.

[0031] For illustration, a given client device 202 can send queries via database execution engine 250 to a database layer comprising one or more databases 290, which can represent a persistence and / or storage layer that can store and / or query database tables. Furthermore, database execution engine 250 can provide the ability to access table storage via an abstract interface to a table adapter, which reduces dependence on specific types of storage and persistence layers, thereby enabling use with different types of storage and persistence layers.

[0032] Database execution engine 250 can be configured to handle different types of databases and their corresponding persistence layers and / or tables. Database execution engine 250 can perform operations including rule-based operations (such as joins and projections, as well as filtering, grouping, multidimensional analysis, etc.) in a way that reduces the processing burden at the database layer. In this way, database execution engine 250 can perform these and other complex operations, while one or more databases 290 can perform simpler operations to reduce the processing burden at one or more databases 290.

[0033] In the example, the database execution engine 250 includes a control unit 270 for determining which Open Table Format (OTF) enhancement levels to apply to queries targeting one or more databases 290. It should be noted that the control unit 270 can be implemented using any suitable combination of hardware (e.g., processing units, programmable logic, application-specific integrated circuits) and software (e.g., program instructions). In the example, the OTF enhancement levels include a first OTF enhancement level that utilizes tags to preserve snapshot versions, stores information about transactions in flight, and prevents concurrent operations from declaring the same snapshot version. The first OTF enhancement level also defines a structure with sub-logs having redo and compensation steps for each transaction step. The OTF enhancement levels also include a second OTF enhancement level that utilizes tags to preserve snapshot versions, stores information about transactions in flight, and prevents concurrent operations from declaring the same snapshot version. The second OTF enhancement level also activates a tag-based conflict and deadlock detection model between multiple tables of database 290. The OTF enhancement levels also include a third OTF enhancement level that implements a global layer that manages multiple table snapshot versions to globally enable multi-version concurrency control. The third OTF enhancement level also implements a global version log, which summarizes the most recent valid versions of multiple tables. In other embodiments, other numbers of OTF enhancement levels may be used and / or functionality may be partitioned among the various OTF enhancement levels in other suitable ways. Further details regarding OTF enhancement levels will be provided in the remainder of this disclosure.

[0034] Now for reference Figure 3This paper describes examples of data lake and lake reservoir architectures based on one or more embodiments of the current topic. A lake reservoir (LH) evolved from a data lake. The LH architecture enables efficient processing of structured data in object storage using query engines. Figure 3 A simplified overview of the different responsibilities of each part of the two architectures is given. (Compared to data lakes) Figure 3 a) Conversely, LH moves data organization (e.g., data placement, version allocation) from external services to object storage ( Figure 3 b).

[0035] A typical use case for LH is machine learning and analytics applications with concurrent writes from various data sources with different requirements. Initial load sizes into object storage can range from several GB to single-digit TB. Workloads often have triple-digit ingestion rates of events ranging from thousands of kB to MB per hour, along with complex analytics queries with double- to triple-digit concurrent users. Required data freshness guarantees range from seconds to days. With LH, analytics applications aim to achieve relatively low storage costs within the data lake while maintaining reasonable performance at the expense of data freshness. Therefore, LH features should be well-suited for such mixed workloads, working around object storage's reliance on immutable files. Existing technologies from other domains cannot be easily applied without adapting to those features.

[0036] Open Table Format (OTF) over object storage is at the heart of the LH architecture. These formats add a separate metadata layer to each table on object storage, organizing data and controlling concurrency from different clients. This enables advanced analytics features across the board, is easily adaptable, provides structure, and is entirely built on open-source formats. However, despite the different metadata layer designs of OTFs, all formats lack the same key features: recovery, complex transactions, and isolation.

[0037] Recovery: Reducing redundant work during aborts. All OTFs define a single table-level synchronization point in their metadata layer as an append log or branch to enable optimistic concurrency control. Therefore, a transaction can only be committed if it successfully passes this bottleneck. All transactions based on the same accessed branch / log must abort because a new commit may have already accessed the same data. However, OTFs do not provide any means to recover or undo failed transactions. This can result in unreferenced files, additional costs, and high latency. Redoing transactions leads to redundant work in the form of additional requests to object storage, especially uploading duplicate data files. For more complex operations, the impact of this missing feature may be exacerbated because they may lack standard OTF concurrency control.

[0038] Complex Transactions: Multi-query and Multi-table Transactions. The OTF design decision, focused on single-query and single-table transactions, results in each operation / sub-operation being published after its execution (a) and the table-level design of metadata (b). (a) It breaks the ACID guarantees of multi-query transactions: when the current design is simply applied, intermediate changes in a multi-query transaction become immediately visible (atomicity) and can potentially be overwritten by other concurrent transactions (consistency, isolation). Furthermore, (b) it prevents multi-table transactions: aborting a transaction in one table can result in a partially committed transaction (consistency) that cannot be rolled back without potentially aborting a newer transaction or deleting committed values ​​(durability, chance of cascading aborts). Therefore, the current approach either has to break isolation or risk starvation from complex queries within complex transactions.

[0039] Isolation: Controlling Global Versioning. Due to the separate metadata layer in OTF, multi-table transactions must interact with the metadata layer of each table individually. This design decision ignores any global information about the state and dependencies of tables, assigning versions at the table level. Under concurrency, other transactions may observe inconsistent combinations of table versions (dirty reads) because some tables may adapt to another transaction's changes earlier than others. Most notably, even limiting a single writer, LH cannot prevent read transactions from observing potentially partially committed multi-table transactions. Global locking can address this issue, but it reduces LH's throughput due to partially enforcing sequential execution. Since concurrent reads are the primary workload of LH, guaranteeing transaction isolation between read and write transactions becomes crucial.

[0040] Now for reference Figure 4 This paper describes an example of metadata layer design for Iceberg tables and incremental lake tables according to one or more embodiments of the current topic. All OTFs essentially add a separate metadata layer to each table on the object storage. By default, when the system accesses a table, it reads and updates the metadata layer to determine and set a consistent version. Therefore, the metadata layer is essential to the design of OTFs and represents the biggest difference from previous architectures.

[0041] Figure 4 The Apache Iceberg metadata layer design in Apache Iceberg shows a top-down structure where all snapshots are defined by a file at the top containing a metadata file. This metadata file contains statistics about each snapshot and redirects to a specific list of inventory items. Figure 4 In table a, snapshot (i) is defined by the metadata file "Meta(i)", which is redirected to list lists "list1", "list2", etc. Each list describes a specific section of the table and is defined by a list file with additional statistics. Figure 4List 1 shows a list of lists, which includes list files "File 1" and "File 2", while "List 2" contains list file "File 3", and so on. Each list file provides references to the various data files and more precise statistics. By assembling all the information from all the list files, the actual table data is fully defined. Figure 4 The layer 'a' displays list files "File 1" and "File 2" pointing to data A, B, and C. Additionally, 'File 3' includes data D and more files. This design requires clients to access this layer from top to bottom to retrieve the complete list of data and to create new snapshots from bottom to top by adding and rewriting files in the tree. Any write operation creates its corresponding data file, and only the most recent list file in its subtree is linked to create its snapshot version. This design can be categorized as an explicit design when the Iceberg metadata layer returns an explicit list of a given snapshot.

[0042] Figure 4 Section b describes the design of the incremental table metadata layer. Snapshots are defined by their corresponding log files and all their previous checkpoints and log files. A single log file contains all file operations to advance a previous snapshot to its own. These operations range from adding or deleting files to adding metadata. Figure 4 b illustrates logs (i), (i-1), and (i-2), each defining its own snapshot. Logs (i-2) and (i-1) add data segments A, B, and C, while log (i) removes segment C and adds other data. Therefore, snapshot (i) contains segments A and B and other data, but not C and D. Incremental lakes avoid sequentially accessing all logs using pointers to the last checkpoint created (typically stored in a separate file) and checkpoints stored in Parquet format. Checkpoints summarize all valid table commands and snapshots at the time of their creation. Figure 4 In b, the last checkpoint is "checkpoint x", which contains logs (i-1), (i-2), and instructions from all previous logs and checkpoints. Therefore, the client must only read "checkpoint x" and "Log(i)" to retrieve snapshot (i).

[0043] To retrieve the latest snapshot, the client issues a list request on the metadata directory, determining the requested snapshot based on the log key. The client then accesses the files and sequentially applies the listed operations to find a consistent table state. Conversely, updating the table is accomplished by appending a new log file to the log directory and ultimately creating a new checkpoint. Because of the design of log instructions rather than an explicit list of files, incremental tables can be categorized as implicitly designed.

[0044] This specification presents several techniques for enabling multi-table and multi-query transactions in data lakes (LH). The first of these techniques (Technique #1) employs tags within the OTF metadata layer to prevent transaction starvation. For LH, tags retain a snapshot version during their transaction execution, thus preventing concurrent operations from claiming the same version. A tag transformation process converts them into the actual part of the OTF metadata layer during commit. Furthermore, tags aid recovery by storing information about transactions in flight. Due to the mixed workloads common in LH, tags appear empty to LH clients like Spark or Trino for seamless integration into existing architectures with OTF. The effect of tags can be compared to pessimistic locking and tombstone tagging.

[0045] Now for reference Figure 5 This describes an example of tag integration based on one or more implementations of the current topic. Figure 5 The placement of the tags shown is not arbitrary. In Iceberg, tags are placed in areas of the manifest list and linked to snapshots. For incremental tables, tags are appended to the log directory. For tag 'M1', both methods are... Figure 5 a and Figure 5 As shown in b. In both options, the flag effectively becomes an exclusive lock to prevent other transactions from claiming the same snapshot version (i+1).

[0046] However, committed transactions should be atomically visible and independent of the tagging technique. Therefore, an alternative commit phase can be used. During the commit, the client gathers and summarizes all relevant information according to the required instructions from the OTF. If necessary, the client constructs some underlying structure, such as an Iceberg manifest list, and finally rewrites the placed tags with all the required instructions. This action effectively makes the entire transaction result available without interfering with the protocol. In the case of abort, the tags are rewritten with an empty manifest list or log entries (tombstone tags).

[0047] Turn now Figure 6 This paper describes examples of physical and logical SAGA structures according to one or more embodiments of the present topic. The SAGA structure is a second technique (i.e., technique #2) for implementing multi-table and multi-query transactions in a data LH. This second technique makes the SAGA pattern applicable to object storage. Following the introduction of the tagged locking system in technique #1, technique #2 focuses on transaction recovery. Technique #2 defines a sub-log structure in a separate working directory, where each transaction step has redo and compensation steps. Such a structure allows for undo and redo within a transaction to any point in time. The included redo and compensation steps can be applied to file-based operations and set up in the LH context. Figure 6Shows for the query "UPDATE SET... WHERE key = <key>"Sub-logs generated in the physical and logical OTF definitions."

[0048] Physical SAGA commands describe explicit changes made to a previous table state, such as adding and removing files. Physical SAGA commands can be compared to log entries in an implicit OTF design (e.g., an incremental table). An example of a physical SAGA command used for the above query is in... Figure 6 As shown in section a. Each instruction references an ID table and uses a specific keyword to identify the instruction's operation. The corresponding compensation steps are stored directly after the delimiter character. Figure 6 The first instruction read in a has a specified key. <key>The file, thus generating the found data file.<path_old> The physical path and file content. The corresponding compensation step simply repeats this operation because the redo operation does not change the entire table state. For the next step, the client uses the newly generated path... <path>The new data file is uploaded below, and the removal of the new data file is defined for the compensation step. Finally, the transaction is removed.<path_old> Add the old data file to the location for compensation.

[0049] Logical SAGA instructions record the operational logic and attempt to greedily apply explicit changes as much as possible. In the example, the design can treat blind appends to tables as non-conflict-free. The client will generate a file and add physical SAGA instructions, even within the logical SAGA structure. Conversely, updates depend on previous transactions and are logically encoded. Figure 6 b shows the logical encoding of an example update query. The first SAGA instruction definition has a given <key>The current file is searched, and the results are stored in the client's buffer. A corresponding compensation step records the old value "old_found" of the buffer to restore the old buffer state. The second SAGA instruction references the current buffer value "found" and sets the referenced file... <path>The new files are merged to generate an updated version. As compensation, it can be... <path>Replace the new file with the original file. The client must ensure that all logical SAGA instructions are executed sequentially before the transaction is committed.

[0050] Undo and redo are operations that utilize the transaction's sub-logs. Undo involves traversing the sub-log structure from the latest to the oldest sub-log and performing compensating steps to restore the previous state. Figure 6 In the example, undo effectively restores the transaction state prior to the update in both log versions. For physical commands, undo in<path_old> Add old files and in <path>The updated files are removed (compensation steps p+1 and p+2). Additionally, the undo mechanism can optionally perform a physical read such as compensation step p. The logical version replaces the path stored in its buffer. <path>Start (compensation step l+1). Internally, this is done by removing at the buffer path. <path>This is achieved by adding files. Finally, the old buffer value...<old_key> This can be stored in the client's buffer for use in the next undo instruction (compensation step 1). In contrast to undo, the redo operation traverses the sub-log structure from oldest to newest file, thus performing the redo step. This effectively causes the client to perform the same operation in the current setup.

[0051] Now for reference Figure 7 This paper describes examples of tag-based conflict and deadlock detection according to one or more embodiments of the present topic. While table-level locking and recovery can prevent conflicts and dependencies within tables (techniques #1 and #2), dependencies between tables still cannot be captured. For multiple tables, deadlocks or inconsistencies may occur when more tags are placed on these tables. Global dependencies can be tracked by introducing novel deadlock and conflict detection by extending tags with known dependencies (technique #1). The novel deadlock and conflict detection strategy proposed in this paper is referred to as technique #3.

[0052] use Figure 7 The example depicted illustrates an example strategy. Transactions T1 and T2 begin working on different tables and place their tags in the corresponding metadata layers. During this time frame, the global ordering of these transactions is concurrent and independent. After some time, T2 requests access to table 1, placing its tag after T1's tag. When initializing the new tag and accessing the table state, T2 recognizes that T1's tag precedes its own. Since this relationship does not violate its own observed dependencies, T2 does not abort. T2 updates its list of all accessed tables by rewriting each tag in the accessed tables. The global ordering between transactions is then T1 before T2. Later, T1 requests access to table 2, placing its tag in table 2's metadata layer, and T1 checks the preceding tags. T1 detects T2's tag and retrieves its data. T1 compares the opened tables with T2's list and checks for potential deadlocks and conflicts. Because T2 also lists Table 1 as accessed and T1 is unaware of the sorting T2 <- T1, T1 infers a potential deadlock. Therefore, T1 releases its old tag in Table 1 by rewriting the old tag with an empty manifest / log and places the new tag "tag T1" in the appropriate metadata layer. After confirming dependency matching (and possibly readjusting some changes in Table 1), T1 continues its operation. If T2 decides to access another table and finds a conflict contradicting its internal observation, T2 will re-evaluate that observation. Here, if the situation changes, T2 checks the origin of the observation (here: Table 1). Depending on the outcome, T2 updates its internal observation or resolves the conflict.

[0053] It is possible that this strategy might cause an abort when releasing the tokens, which can be resolved by using technique #2 to perform a full transaction restart using the tokens already acquired. Similar to technique #1, a standard commit rewrites all tokens with the corresponding changes to all tokens.

[0054] Turn now Figure 8 This describes an example of a global version log based on one or more embodiments of the current topic. Following the introduction of techniques #1-3 for snapshot preservation, recovery, and detection of global deadlocks, technique #4 includes a novel global layer that can be defined for object storage, where the novel global layer manages all table snapshot versions to globally enable Multi-Version Concurrency Control (MVCC-like) behavior. This can be referred to as a file-based layered global version log, where the file-based layered global version log summarizes the latest valid versions of all tables. Therefore, this technique eliminates the need for transactions to explicitly or implicitly retrieve their versions individually from each table and provides greater control over tracking global relationships and their dependencies. By shifting version allocation to a global design, snapshot versions can be atomically advanced to guarantee isolation between concurrent transactions.

[0055] In the example, the new global version log 820 handles all table versions. If a client accesses a table, it first reads global version log 820 and retrieves its global snapshot using a single GET request. Afterward, the client is bound to the received version and can only use table versions equal to or less than the specified version. Figure 8 The example in the example shows the global version log 820 tracking four tables and their latest snapshot versions. T1 and T2 both start their execution and retrieve version combinations (Table 1, version 5), (Table 2, version 3), (Table 3, version 2), and (Table 4, version 6) from the global version log.

[0056] Global Version Log 820 introduces a file-based verification phase for commits, which checks the compatibility of transactions with the current global version, similar to MVCC. If verification succeeds, the transaction performs a table-level commit for each accessed table before atomically announcing a new snapshot. Transaction verification begins by checking for version changes to the transaction itself and copying them to the shared key on object storage 810. If the transaction succeeds (by checking that the file has not been overwritten by concurrent operations), it continues by reading the latest global version log 820. Afterward, the transaction iterates over its own transaction version log and compares version numbers. A read sub-operation is compatible if its own version equals the global log version. However, write sub-operations increment the version in the transaction log. These entries are compatible only if the difference between the old version and the proposed version equals the number of changes performed by the transaction. Unaccessed or newly created tables are not included in either global version log 820 or the transaction log. Both cases are automatically compatible, and more information is appended to global version log 820 during commit. As an additional optimization, rules are introduced that allow read-only transactions to be compatible with lower list values ​​or even skip version comparisons altogether. A transaction is considered compatible if all its operations are confirmed to be compatible with global version log 820 during the verification process.

[0057] exist Figure 8 In the example, T1 reads tables 1 and 2 and writes to table 3, while T2 reads table 4, writes to table 3, and creates table 5. Their respective transaction logs contain the current version of the tables read and the target version of the tables written. Both transactions initiate validation and attempt to copy their transaction logs to the shared path. In this example, T1 is lost and must be aborted or recovered. T2 continues to compare global version log 820 with its transaction log. Since T2 does not access tables 1 and 2, they are automatically compatible with the changes made by the transactions. The read statement for table 4 is the same version as that in global version log 820, so the read of table 4 is compatible. The write set contains reference version 3 (2+1) of table 3 and version 1 of table 5. Both changes are compatible, claiming a higher version tracked in global version log 820. Therefore, T2 is compatible and is tiered for commit. T2 triggers a separate commit on all tables and updates global version log 820. After each separate commit operation succeeds, the transaction deletes its copy in the shared transaction log directory to enable the next transaction validation. In contrast to compatible changes, if verification fails and the transaction is deemed incompatible, the transaction must be aborted or a recovery mechanism must be triggered.

[0058] Now that each individual technique has been described, a cumulative LakeVilla prototype is introduced. The cumulative LakeVilla prototype assembles various techniques to provide individual solutions for missing features: Level 0 enables the recovery of transactions from concurrent changes (techniques #1 and #2). Level 1 extends transaction logic to support multi-table and multi-query operations (techniques #1 and #3). Level 2 guarantees that transactions are always isolated (technique #4). Each level is isolated, meaning that levels can be applied without a higher or lower level. Furthermore, the LakeVilla prototype can be seamlessly applied to OTFs such as Apache Iceberg and Delta Lake without reformatting or reconfiguring the entire system. Each level will now be discussed separately.

[0059] Lakevilla Level 0: Recovery - A Sign of Preventing Starvation. A key challenge in preventing aborts when operating on concurrent transactions is adapting to changes in snapshot versions at the metadata layer. By default, implicit and explicit metadata designs apply optimistic concurrency control, moving conflict checking to transaction commit. This approach assumes that most operations do not conflict. However, OTF operates at the table level, meaning that all concurrent write operations on the same table will conflict without fine-grained conflict management. Therefore, Technique #1 can be used to reserve a snapshot ID at the start of a transaction and invalidate it in the event of an abort. Furthermore, Technique #2 provides more information on conflict resolution and partial adaptation to concurrency changes.

[0060] Table-level conflict detection. To determine whether to commit or abort, Lakevilla Level 0 uses flags placed in the table's metadata layer. For concurrent transactions, conflict detection is triggered as soon as the flag of the previous transaction is replaced. The corresponding client accesses the contents of the new log and checks whether the previous transaction and its own transaction have modified the same file. If a conflict occurs, recovery is achieved using the SAGA structure. Depending on the location of the corresponding flag, the transaction's SAGA log record type can be adapted: if the transaction did not observe any previous flags during its placement in the metadata layer, the transaction uses physical instructions within its SAGA structure; otherwise, it uses logical instructions. At commit, physical instructions can be applied directly, while logical instructions must be redone during commit. This hybrid approach of techniques #1 and #2 enables higher throughput in the metadata layer and transforms flags from exclusive locks to shared locks.

[0061] LakeVilla Level 1: Complex Transactions - Multi-query and Multi-table Transactions in LakeVilla. Building upon LakeVilla Level 0, multi-query and multi-table transactions can be implemented for LH. For multi-query transactions on the same table, the tagging idea (Technique #1) can be reused, but the tagging model can be extended with Technique #3. As before, LakeVilla has a tag to lock the table until the entire multi-query transaction is processed and rewritten with a real manifest / log to unlock it. Unfortunately, reapplying the tagging idea only partially implements complex transactions because multi-table transactions break global dependencies due to the separate metadata layers of all OTF tables. A single table's metadata layer will not recognize any dependencies between tables, leading to inconsistencies, deadlocks, and partial aborts. To avoid these problems, transactions must declare their global presence and check table compatibility. LakeVilla Level 1 exposes this information by allowing in-flight transactions to declare access to new tables with multiple tags and using tag-based conflict detection. If a client detects a deadlock, the client resolves the issues specified by Technique #3. Furthermore, when a transaction succeeds on all tables, a commit triggers all table-level commits required by the transaction by globally replacing all flags and commits.

[0062] LakeVilla Level 2: Isolation – The Need for a Global Lake Reservoir Layer. To fully guarantee isolation for any transaction, changes made by a transaction must become available globally simultaneously. As mentioned earlier, using a table-level metadata layer in the current lake reservoir design does not guarantee simultaneous commits across multi-table OTF settings. Therefore, a global layer (Technique #4) can be introduced to manage commits and global dependencies. This change guarantees that versions are globally consistent at all times. Furthermore, LakeVilla Level 2 implements version allocation and verification steps from MVCC classes on the object storage. This approach minimizes costs and avoids slow requests, such as those to a list in the object storage.

[0063] Modified Read and Write Transactions – Write transactions follow the strategy of proposed technique #4. Write transactions retrieve a globally consistent version by reading the global version log and perform all operations independently. Upon commit, the transaction validates its changes and atomically updates the global version log. If the changes are incompatible, the write transaction triggers a recovery mechanism (e.g., technique #2) or must abort. To enable read transactions to retrieve a globally consistent version, read transactions read the global version log and skip snapshot allocations from their native OTF protocol. This effectively reduces the number of requests to the metadata layer of a single table and replaces expensive requests (e.g., lists used for implicit design).

[0064] Now for reference Figure 9 The diagram illustrates an example of a system 900 according to one or more implementations of the present topic. In system 900, clients can store data in object storage in an open format. Furthermore, system 900 can implement LakeVilla, an extension that implements three missing features at a composable level. In the example, feature 0 supports complex transactions by implementing a conflict and deadlock detection model across all tables. The conflict and deadlock detection model uses tags (e.g., M0, M1, M2) to create a global ordering among concurrent transactions. Feature 1 supports recovery by implementing logical snapshot version tags, where logical snapshot version tags have transaction sub-logs and physical and logical undo and redo information to define a greedy transaction model. Tags point to subdirectory structures that enable the recovery of changes to an earlier point in time for an ongoing transaction. Tags and sub-logs are stored in OTF metadata layers 920A-920B for tables A and B, respectively. Feature 2 supports isolation by introducing a global version log 910, which examines global relationships between tables and acts as a global layer on the object storage.

[0065] System 900 implements the previously described features to address the challenges of supporting multi-table, multi-query transactions in existing systems. Each client can utilize these features on a feature-by-feature basis. However, clients do not need to utilize these features when accessing data in object storage. As shown in the figure, client C0 implements features 0, 1, and 2, and client C0 interacts with object storage via a file-based global version log 910. Furthermore, client C1 implements features 0 and 2, and client C1 interacts with object storage via a file-based global version log 910. On the other hand, client C2 only implements feature 1, and client C2 bypasses the file-based global version log 910 and interfaces directly with the data in object storage.

[0066] Now go to Figure 10 This describes a process, according to one or more embodiments of the present subject, for determining which levels to apply to queries targeting a data lake. At the start of the process, transactions targeting a data lake employing Open Table Format (OTF) are detected (box 1005). In the example, the transaction targets multiple tables and includes multiple queries. In other examples, the transaction may target only a single table and / or the transaction may include only a single query. Next, the database execution engine (e.g., ...) ... Figure 2 The database execution engine (250) determines which OTF enhancement levels are applied to transactions (Box 1010). In the example, the OTF enhancement levels include a first OTF enhancement level, which utilizes marking to preserve snapshot versions, stores information about transactions in flight, and prevents concurrent operations from declaring the same snapshot version. The first OTF enhancement level also defines a sub-log structure with redo and compensation steps for each transaction step. The OTF enhancement level also includes a second OTF enhancement level, which utilizes marking to preserve snapshot versions, stores information about transactions in flight, and prevents concurrent operations from declaring the same snapshot version. The second OTF enhancement level also activates a marking-based conflict and deadlock detection model between multiple tables in the target database. The OTF enhancement level also includes a third OTF enhancement level that implements a global layer that manages multiple table snapshot versions to globally enable multi-version concurrency control. The third OTF enhancement level also implements a global version log, which summarizes the most recent valid versions of multiple tables.

[0067] The database execution engine offers the flexibility to utilize different OTF enhancement levels for different queries. For example, the engine can utilize the first OTF enhancement level for the first transaction, the second for the second, the third for the third, both for the fourth, and both for the fifth. It can also choose not to utilize any OTF enhancement level for the sixth transaction, and so on. In other words, the engine can mix and match different combinations of OTF enhancement levels for different queries based on how they are constructed, allowing these enhancement levels to be applied or not applied on a transaction-by-transaction basis.

[0068] If the database execution engine determines that the first OTF enhancement level should be applied to a transaction (condition box 1015, "Yes" branch), the database execution engine applies the first OTF enhancement level to the transaction by utilizing flags to preserve snapshot versions, storing information about transactions in flight, and preventing concurrent operations from declaring the same snapshot version (box 1020). The database execution engine also applies the first OTF enhancement level to the transaction by defining a structure with sub-logs having redo and compensation steps for each transaction step (box 1025).

[0069] If the database execution engine determines to apply the second OTF enhancement level to a transaction (condition box 1030, "Yes" branch), the database execution engine applies the second OTF enhancement level to the transaction by leveraging flags to preserve snapshot versions, storing information about transactions in flight, and preventing concurrent operations from declaring the same snapshot version (box 1035). The database execution engine also applies the second OTF enhancement level to the transaction by activating a flag-based conflict and deadlock detection model between multiple tables in the data lake database (box 1040).

[0070] If the database execution engine determines that the third OTF enhancement level should be applied to a transaction (condition box 1045, "Yes" branch), the database execution engine applies the third OTF enhancement level to the transaction by implementing a global layer that manages multiple table snapshot versions to globally enable multi-version concurrency control (box 1050). The database execution engine also applies the second OTF enhancement level to the transaction by activating a tag-based conflict and deadlock detection model between multiple tables in the data lake (box 1055).

[0071] Next, the database execution engine enables the transaction to be processed using any of the OTF enhancement levels that have been determined to be applied to the transaction (box 1060). The database execution engine then returns the result of the transaction processing to the computing device of the requesting client (box 1065). After box 1065, method 1000 can terminate.

[0072] Now for reference Figure 11 This describes a process for implementing nested metadata layers in an object storage according to one or more embodiments of the present topic. The object storage implements nested metadata layers, wherein the nested metadata layers include at least a global metadata layer (e.g., Figure 9 The global version log 910) and table metadata layers (e.g., metadata layers 920A-920B) (Box 1105). For each transaction target data in object storage, the control unit (e.g., Figure 2 The control unit 270 determines whether a transaction interacts with each of the nested metadata layers (box 1110). For transactions requiring global consistency, the control unit causes the transaction to traverse the global metadata layer, which includes a global version log for maintaining global consistency (box 1115). For transactions requiring global ordering among concurrent transactions, the control unit causes the transaction to traverse the table metadata layer of each table accessed, which includes flags for creating global ordering and sub-logs for enabling undo and redo operations (box 1120). The control unit then causes other transactions to bypass the global metadata layer and / or the table metadata layer on a transaction-by-transaction basis (box 1125). After box 1125, method 1100 may terminate.

[0073] In some implementations, the current topic can be configured to be implemented in system 1200, such as Figure 12A As shown. System 1200 may include processor 1210, memory 1220, storage device 1230, and input / output device 1240. Each component (e.g., processor 1210, memory 1220, storage device 1230, I / O device 1240) may be interconnected using system bus 1250. Processor 1210 may be configured to process instructions for execution within system 1200. In some embodiments, processor 1210 may be a single-threaded processor. In alternative embodiments, processor 1210 may be a multi-threaded processor. Processor 1210 may also be configured to process instructions stored in memory 1220 or storage device 1230, including receiving or transmitting information via input / output device 1240. Memory 1220 may store information within system 1200. In some embodiments, memory 1220 may be a computer-readable medium. In alternative embodiments, memory 1220 may be a volatile memory cell. In still other embodiments, memory 1220 may be a non-volatile memory cell. Storage device 1230 may provide mass storage for system 1200. In some embodiments, storage device 1230 may be a computer-readable medium. In alternative embodiments, storage device 1230 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, non-volatile solid-state memory, or any other type of storage device. Input / output device 1240 may be configured to provide input / output operations for system 1200. In some embodiments, input / output device 1240 may include a touchscreen display capable of displaying a graphical user interface.

[0074] Figure 12B Depicting ( Figure 1 This is an example implementation of system 100. System 100 can be implemented using various physical resources 1280, such as at least one or more hardware servers, at least one storage, at least one memory, at least one network interface, etc. System 100 can also be implemented using infrastructure, which, as described above, may include at least one operating system 1282 and at least one hypervisor 1284 for the physical resources 1280 (which can create and run at least one virtual machine 1286). For example, each multitenant application can run on the corresponding virtual machine 1286.

[0075] The systems and methods disclosed herein can be embodied in various forms, including, for example, data processors, computers that also include databases, digital electronic circuits, firmware, software, or combinations thereof. Furthermore, the aforementioned features and other aspects and principles of the embodiments of this disclosure can be implemented in various environments. Such environments and related applications can be specifically constructed to perform various processes and operations according to the disclosed embodiments, or they can include general-purpose computers or computing platforms that are selectively activated or reconfigured by code to provide necessary functionality. The processes disclosed herein are not inherently associated with any particular computer, network, architecture, environment, or other device, and can be implemented through suitable combinations of hardware, software, and / or firmware. For example, various general-purpose machines can be used with programs written according to the teachings of the disclosed embodiments, or it may be more convenient to construct dedicated devices or systems to perform the required methods and techniques.

[0076] While ordinal numbers such as first, second, etc., can relate to order in some cases, as used in documentation, ordinal numbers do not necessarily imply order. For example, ordinal numbers may be used simply to distinguish one item from another. For instance, to distinguish the first event from the second event, without implying any chronological order or fixed frame of reference (so that the first event in one paragraph of the specification may be different from the first event in another paragraph of the specification).

[0077] The foregoing description is intended to illustrate, and not limit, the scope of the invention, which is defined by the appended claims. Other embodiments are within the scope of the appended claims.

[0078] These computer programs (also referred to as programs, software, software applications, applications, components, or code) include program instructions (i.e., machine instructions) for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the term "machine-readable storage medium" refers to any computer program product, apparatus, and / or device for providing machine instructions and / or data to a programmable processor, such as disks, optical disks, memories, and programmable logic devices (PLDs), including machine-readable storage media that receive program instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor. Machine-readable storage media may store such program instructions non-transitory, such as, for example, non-transitory solid-state memory or magnetic hard disk drives or any equivalent storage medium. Machine-readable storage media may alternatively or additionally store such machine instructions transiently, such as processor caches or other random access memory associated with one or more physical processor cores.

[0079] To provide interaction with the user, the subjects described herein can be implemented on a computer having a display device for displaying information to the user (such as, for example, a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) and a keyboard and pointing device (such as, for example, a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback, such as, for example, visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0080] The subject matter described herein can be implemented in a computing system that includes backend components (such as, for example, one or more data servers), or middleware components (such as, for example, one or more application servers), or frontend components (such as, for example, one or more client computers having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium, such as a communication network. Examples of communication networks include, but are not limited to, local area networks ("LANs"), wide area networks ("WANs"), and the Internet.

[0081] A computing system may include clients and servers. Clients and servers are typically, but not exclusively, geographically separated and usually interact through a communication network. The client-server relationship is established by means of computer programs running on the respective computers and having a client-server relationship with each other.

[0082] In the foregoing description and claims, phrases such as "at least one of..." or "one or more of..." may appear, followed by a linked list of elements or features. The term "and / or" may also appear in a list of two or more elements or features. Unless otherwise implicitly or explicitly contradicted by the context in which it is used, such phrases are intended to mean any one of the individually listed elements or features, or any one of the recited elements or features in combination with any of the other recited elements or features. For example, the phrases "at least one of A and B;", "one or more of A and B;", and "A and / or B" are each intended to mean "A alone, B alone, or A and B together". Similar interpretations are also intended for lists comprising three or more items. For example, the phrases "at least one of A, B, and C;", "one or more of A, B, and C;", and "A, B, and / or C" are each intended to mean "A alone, B alone, C alone, A and B together, A and C together, B and C together, or A and B and C together". The use of the term "based on" in the foregoing and claims is intended to mean "at least partially based on," so that features or elements not listed are also permissible.

[0083] In view of the above-described embodiments of the subject matter, this application discloses the following list of examples, wherein a feature of a single example or a combination thereof, and optionally in combination with one or more features of one or more other examples, is also an additional example falling within the disclosure of this application:

[0084] Example 1: A system comprising: at least one processor; and at least one memory storing instructions, the instructions causing operations when executed by the at least one processor, the operations including: detecting a transaction targeting an object storage employing Open Table Format (OTF); determining which of a plurality of OTF enhancement levels to apply to the transaction; applying one or more OTF enhancement levels to the transaction based on the determination; causing the transaction to be processed using the applied one or more OTF enhancement levels; and returning the result of the transaction processing to a computing device.

[0085] Example 2: The system based on Example 1, wherein applying the first OTF enhancement level includes: using tags to preserve snapshot versions, storing information about transactions in flight, and preventing concurrent operations from declaring the same snapshot version; and defining a structure with sub-logs having redo and compensation steps for each transaction step.

[0086] Example 3: A system based on any one of Examples 1-2, wherein applying the second OTF enhancement level includes: using tags to preserve snapshot versions, storing information about transactions in flight, and preventing concurrent operations from declaring the same snapshot version; and activating a tag-based conflict and deadlock detection model between multiple tables in object storage.

[0087] Example 4: A system based on any of Examples 1-3, where the application of the third OTF enhancement level includes implementing a global layer that manages multiple table snapshot versions to globally enable multi-version concurrency control.

[0088] Example 5: A system based on any of Examples 1-4, where applying the third OTF enhancement level also includes implementing a global version log that summarizes the most recent valid versions of multiple tables.

[0089] Example 6: A system based on any of Examples 1-5, where each OTF enhancement level can be applied independently of other OTF enhancement levels among a plurality of OTF enhancement levels.

[0090] Example 7: A system based on any of Examples 1-6, wherein applying the first OTF enhancement level also includes storing the first tag in the metadata layer while the first transaction is in flight.

[0091] Example 8: A system based on any of Examples 1-7, wherein the operation further includes converting the first token into a structure within the metadata layer during the commit of the first transaction.

[0092] Example 9: A system based on any of Examples 1-8, where transactions target multiple tables.

[0093] Example 10: A system based on any of Examples 1-9, where a transaction comprises multiple queries.

[0094] Example 11: A computer-implemented method comprising: detecting a transaction targeting an object storage employing Open Table Format (OTF); determining which of a plurality of OTF enhancement levels should be applied to the transaction; applying one or more OTF enhancement levels to the transaction based on the determination; causing the transaction to be processed using the applied one or more OTF enhancement levels; and returning the result of the transaction processing to a computing device.

[0095] Example 12: The computer-implemented method according to Example 11, wherein applying the first OTF enhancement level includes: using tags to preserve snapshot versions, storing information about transactions in flight, and preventing concurrent operations from declaring the same snapshot version; and defining a structure with sub-logs having redo and compensation steps for each transaction step.

[0096] Example 13: A computer-implemented method according to any one of Examples 11-12, wherein applying the second OTF enhancement level includes: using tags to preserve snapshot versions, storing information about transactions in flight, and preventing concurrent operations from declaring the same snapshot version; and activating a tag-based conflict and deadlock detection model between multiple tables in object storage.

[0097] Example 14: A computer-implemented method according to any of Examples 11-13, wherein applying a third OTF enhancement level includes implementing a global layer that manages multiple table snapshot versions to globally enable multi-version concurrency control.

[0098] Example 15: A computer-implemented method according to any of Examples 11-14, wherein applying the third OTF enhancement level also includes implementing a global version log that aggregates the most recent valid versions of multiple tables.

[0099] Example 16: A computer-implemented method according to any of Examples 11-15, wherein each OTF enhancement level can be applied independently of the other OTF enhancement levels among a plurality of OTF enhancement levels.

[0100] Example 17: A computer-implemented method according to any one of Examples 11-16, wherein applying the first OTF enhancement level further includes storing the first tag in the metadata layer while the first transaction is in flight.

[0101] Example 18: A computer-implemented method according to any one of Examples 11-17 further includes converting the first token into a structure within the metadata layer during the commit of the first transaction.

[0102] Example 19: A computer-implemented method according to any of Examples 11-18, wherein a transaction targets multiple tables and includes multiple queries.

[0103] Example 20: A non-transitory computer-readable storage medium for storing instructions, which, when executed by at least one data processor, cause operations including: detecting a transaction targeting an object storage employing Open Table Format (OTF); determining which of a plurality of OTF enhancement levels should be applied to the transaction; applying one or more OTF enhancement levels to the transaction based on the determination; causing the transaction to be processed using the applied one or more OTF enhancement levels; and returning the result of the transaction processing to a computing device.

[0104] The embodiments described above do not represent all embodiments consistent with the subject matter described herein. Rather, they are merely examples of aspects consistent with the described subject matter. Although some variations have been described in detail above, other modifications or additions are possible. In particular, further features and / or changes may be provided in addition to those set forth herein. For example, the above embodiments may be applicable to various combinations and sub-combinations of the disclosed features and / or combinations and sub-combinations of several other features disclosed above. Furthermore, the logical flows depicted in the drawings and / or described herein do not necessarily require the specific order or sequence shown to achieve the desired results. Other embodiments are within the scope of the appended claims.< / path> < / path> < / path> < / path> < / path> < / key> < / path> < / key> < / key>

Claims

1. A system comprising: At least one processor; and At least one memory storing instructions that, when executed by the at least one processor, cause an operation including: Detect transactions targeting object storage that uses the Open Table Format (OTF) format; Determine which of the multiple OTF enhancement levels should be applied to the transaction; Based on the determination, one or more OTF enhancement levels will be applied to the transaction; This enables the transaction to be processed using one or more applied OTF enhancement levels; and The results of the transaction are returned to the computing device.

2. The system of claim 1, wherein applying the first OTF enhancement level comprises: Use tags to preserve snapshot versions, store information about in-flight transactions, and prevent concurrent operations from declaring the same snapshot version; and Define a structure with sub-logs for redo and compensation steps for each transaction step.

3. The system of claim 2, wherein applying the second OTF enhancement level comprises: Use tags to preserve snapshot versions, store information about in-flight transactions, and prevent concurrent operations from declaring the same snapshot version; and Activate a tag-based conflict and deadlock detection model among multiple tables in the object storage.

4. The system of claim 3, wherein applying the third OTF enhancement level includes implementing a global layer that manages multiple table snapshot versions to globally enable multi-version concurrency control.

5. The system according to claim 4, wherein applying the third OTF enhancement level further includes implementing a global version log, the global version log summarizing the most recent valid versions of multiple tables.

6. The system of claim 5, wherein each OTF enhancement level can be applied independently of the other OTF enhancement levels among the plurality of OTF enhancement levels.

7. The system of claim 6, wherein applying the first OTF enhancement level further includes storing the first tag in the metadata layer while the first transaction is in flight.

8. The system of claim 7, wherein the operation further includes converting the first token into a structure within the metadata layer during the commit of the first transaction.

9. The system according to claim 1, wherein, The transaction targets multiple tables.

10. The system according to claim 1, wherein, The transaction includes multiple queries.

11. A computer-implemented method, comprising: Detect transactions targeting object storage that uses the Open Table Format (OTF) format; Determine which of the multiple OTF enhancement levels should be applied to the transaction; Based on the determination, one or more OTF enhancement levels will be applied to the transaction; This enables the transaction to be processed using one or more OTF enhancement levels that are applied; and The results of the transaction are returned to the computing device.

12. The computer-implemented method of claim 11, wherein applying the first OTF enhancement level comprises: Use tags to preserve snapshot versions, store information about in-flight transactions, and prevent concurrent operations from declaring the same snapshot version; and Define a structure with sub-logs for redo and compensation steps for each transaction step.

13. The computer-implemented method of claim 12, wherein applying the second OTF enhancement level comprises: Use tags to preserve snapshot versions, store information about in-flight transactions, and prevent concurrent operations from declaring the same snapshot version; and Activate a tag-based conflict and deadlock detection model among multiple tables in the object storage.

14. The computer-implemented method of claim 13, wherein applying the third OTF enhancement level includes implementing a global layer that manages multiple table snapshot versions to globally enable multi-version concurrency control.

15. The computer-implemented method of claim 14, wherein applying the third OTF enhancement level further includes implementing a global version log, the global version log summarizing the most recent valid versions of multiple tables.

16. The computer-implemented method of claim 15, wherein each OTF enhancement level can be applied independently of the other OTF enhancement levels among the plurality of OTF enhancement levels.

17. The computer-implemented method of claim 16, wherein applying the first OTF enhancement level further includes storing the first tag in the metadata layer while the first transaction is in flight.

18. The computer-implemented method of claim 17, further comprising converting the first token into a structure within the metadata layer during the commit of the first transaction.

19. The computer-implemented method according to claim 11, wherein, The transaction targets multiple tables, and the transaction includes multiple queries.

20. A non-transitory computer-readable storage medium for storing instructions, said instructions causing operations when executed by at least one data processor, said operations comprising: Detect transactions targeting object storage that uses the Open Table Format (OTF) format; Determine which of the multiple OTF enhancement levels should be applied to the transaction; Based on the determination, one or more OTF enhancement levels will be applied to the transaction; This enables the transaction to be processed using one or more OTF enhancement levels that are applied; and The results of the transaction are returned to the computing device.

Images