A cross-domain transaction concurrency control method based on lake-warehouse fusion architecture

Abstract

This invention provides a cross-domain transaction concurrency control method based on a lakehouse fusion architecture, belonging to the field of transaction concurrency control technology. This invention sends each transaction to a lakehouse server, assigns a globally unique identifier (TID) to each transaction based on the local clock, sorts all transactions across all lakehouse servers for the current time period to obtain a global order, performs conflict detection and transaction execution, and then performs commit verification to obtain a global snapshot table. This invention provides a cross-domain transaction concurrency control method based on a lakehouse fusion architecture. Transactions within a local lakehouse do not require wide area network (WAN) round-trip communication, while cross-lakehouse transactions only require one WAN round-trip communication, reducing overall communication latency by one round. Simultaneously, transaction segmentation based on time intervals allows transaction latency to be amortized among all transactions within an epoch, reducing the average transaction latency. The optimistic asynchronous execution and speculative read strategies used improve data visibility within the same epoch.

Description

Technical Field

[0001] This invention relates to the field of transaction concurrency control technology, specifically a cross-domain transaction concurrency control method based on a lakeware fusion architecture. Background Technology

[0002] Transaction processing in a lakeware converging architecture is a key technology for ensuring data consistency and integrity. A transaction refers to a series of operations executed as a single logical unit. For example, when performing real-time sales data analysis on an e-commerce platform under a lakeware converging architecture, at least the following operations on the data lake and data warehouse are involved: writing sales records every minute to log files in the data lake in real time; cleaning sales records to remove invalid data (such as duplicate records or incorrect formats); aggregating sales data by hour to generate an "hourly sales" analysis table and saving it to the data warehouse; updating the metadata table and marking the "hourly sales" analysis table as queryable. Under normal circumstances, all the above operations proceed smoothly, ultimately completing a full real-time data analysis task, and all data states and metadata related to the task are successfully updated. However, if a problem occurs in any step (e.g., the log file is not completely written or the metadata table update fails), the entire operation will fail. In the event of failure, the system must ensure that the state of all data and metadata remains unchanged from the state before the commit, and transactions in a lakeware converging architecture are precisely the technology used to ensure the smoothness and predictability of operations in such situations.

[0003] Polaris is the core distributed SQL engine for Azure Synapse Analytics, designed for large-scale data processing in cloud environments. Polaris' transaction processing employs a distributed architecture, encompassing multiple stages including transaction initiation, read / write operations, concurrency control, and storage optimization to achieve data consistency and efficiency. The client first initiates a transaction request; the SQL Front End (FE) creates the root transaction and assigns a unique transaction ID. Read operations, based on snapshot isolation semantics, are performed by capturing snapshots of the database tables (generated from the Manifest files of committed transactions), and the SQL Back End (BE) executes queries within the snapshot to generate results. In write operations, the system generates corresponding Parquet data files or deletion vector files based on insert, update, or delete operations and writes the change records to the transaction Manifest file. During the transaction commit phase, the serialization order of transactions is ensured by acquiring commit locks, and potential conflicts are detected during the verification process; if no conflicts are found, all changes are committed; otherwise, the system rolls back to the initial state. Simultaneously, Polaris effectively avoids problems such as dirty reads, non-repeatable reads, and phantom reads through Optimistic Concurrency Control (OCC) and snapshot isolation semantics.

[0004] NoisePage is an in-memory database management system (DBMS) that supports high-performance transaction processing workloads and optimizes data export processes with external data analysis tools. NoisePage's transaction processing mechanism combines optimistic concurrency control (OCC) and multi-version concurrency control (MVCC). By assigning timestamp pairs (start and commit) to each transaction, it maintains a version chain to support concurrent operations while avoiding write-write conflicts to reduce rollback risks. When a transaction commits, the system assigns a commit timestamp and updates the log manager to ensure the persistence of transaction changes; rollbacks utilize undo records to recover data. A pre-write log and checkpoint mechanism ensure durability; transactions are marked as committed after the log is written to disk. Garbage collection cleans up invalid records by pruning the version chain and releasing memory, ensuring efficient utilization of system resources and achieving consistent, high-performance transaction processing.

[0005] The lake warehouse converged architecture uses open data format object storage. Existing methods have the problem of high overhead in consistency guarantee mechanism, and transaction concurrency control does not perform well on the lake warehouse architecture.

[0006] Polaris uses SQLDB for transaction management, increasing system complexity. The multi-level Manifest files and WriteSets table structure also increase the consistency requirements after transaction execution, necessitating more synchronization, coordination, and verification. If the Polaris system is distributed across multiple data centers, cross-datacenter consistency and transaction processing can become even more complex, requiring additional mechanisms to ensure data consistency in a distributed environment. Under high-concurrency writes, the need to coordinate multiple concurrent write operations may lead to some latency. Furthermore, although the system is designed as a stateless computing architecture, error recovery and rollback processes may still face challenges in certain extreme cases (such as large-scale node failures).

[0007] NoisePage suffers from write amplification. Specifically, when updating variable-length values, the entire value buffer needs to be copied and the offset array updated, increasing the overhead of write operations. In summary, NoisePage sacrifices transaction processing speed by using the Arrow data format to improve data export efficiency. The complex mechanism and specific data format not only result in worse throughput than existing methods in update-intensive transaction scenarios, but also contribute to this issue. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the present invention aims to propose a cross-domain transaction concurrency control method based on a lakeware fusion architecture, comprising:

[0009] Step 1: Send each transaction to a lake warehouse server. Each transaction contains multiple operations and has a corresponding read set, write set, and parameters. Based on the local clock, assign a globally unique identifier (TID) to each transaction and write the read set, write set, and parameters of the transaction to the lake warehouse server corresponding to the transaction. Obtain the underlying lake warehouse snapshot at the time corresponding to the globally unique identifier (TID). The underlying lake warehouse snapshot includes the time corresponding to the globally unique identifier (TID) and the status of the data in the lake warehouse server.

[0010] Step 2: Sort all transactions on all lake warehouse servers for the current time period to obtain the global order;

[0011] Step 2.1: For the current lake warehouse server, obtain all transactions within the current time period. There are read-after-write dependencies and write-after-read dependencies between transactions;

[0012] Step 2.2: Determine whether the transaction crosses a lake warehouse. If the transaction does not cross a lake warehouse, execute the transaction directly. If the transaction crosses a lake warehouse, obtain all transactions that cross lake warehouses and execute Step 2.3.

[0013] Step 2.3: Sort all transactions across lake warehouses to obtain a sequence representing the partial order relationship. Based on the sequence representing the partial order relationship, generate a global order.

[0014] Step 3: For each transaction in the global order, determine whether there are conflicts among all operations within the transaction and between transactions. When there are no conflicts among all operations within the transaction and between transactions, execute all transactions in the global order according to the global order and the underlying lakehouse snapshot to obtain visible version data. When there are conflicts among all operations within the transaction and between transactions, execute the operations within the transaction serially according to the underlying lakehouse snapshot and execute all transactions in parallel to obtain visible version data. When there are conflicts among all operations within the transaction and between transactions, pass the transaction and retain its unique identifier TID, and add it to the transactions in the next time period.

[0015] Step 4: In global order, determine whether the visible version data of the previous transaction of the current transaction can be converted into committed version data. If there are no conflicts in all operations within the transaction and no conflicts between transactions, or if there are conflicts in all operations within the transaction and no conflicts between transactions, it indicates that the visible version data of the previous transaction of the current transaction can be converted into committed version data. In global order, write the visible data of the transaction back to the snapshot. If there are conflicts in all operations within the transaction and conflicts between transactions, it indicates that the visible version data of the previous transaction of the current transaction cannot be converted into committed version data. Then, take the previous transaction of the current transaction as the current transaction and return to execute: determine whether the visible version data of the previous transaction of the current transaction can be converted into committed version data. After the above processing, all transactions in the global order obtain the first snapshot table.

[0016] Step 5: Write the execution results of the transactions that do not cross lake warehouses in Step 2.2 to the global snapshot, and combine them with the first snapshot table obtained in Step 4 to generate a global snapshot table.

[0017] Optionally, step 2.3 specifically includes:

[0018] Step 2.3.1: Generate a local dependency graph based on the read-after-write and write-after-read dependencies that exist between transactions;

[0019] Step 2.3.2: Based on the topological sorting algorithm, generate a position for each transaction in the local dependency graph to obtain a sequence representing the partial order relationship;

[0020] Step 2.3.3: Generate a global order based on the sequences representing partial order relations in all lake warehouse servers.

[0021] Optionally, step 2.3.1 specifically includes:

[0022] Step 2.3.1.1: Construct an initial dependency graph, which contains multiple nodes, each node corresponding to one transaction across all lake warehouse transactions;

[0023] Step 2.3.1.2: For each transaction in all cross-lake warehouse transactions, and for each data item in the read set of that transaction, update the initial dependency graph to obtain the first dependency graph based on the read-after-write dependency and write-after-read dependency relationships between transactions.

[0024] Step 2.3.1.3: For each transaction in all cross-lake warehouse transactions, and for each data item in the write set of that transaction, update the first dependency graph to obtain the second dependency graph based on the read-after-write dependency and write-after-read dependency relationships between transactions, and further update the second dependency graph to obtain the local dependency graph.

[0025] Optionally, step 2.3.1.2 specifically includes:

[0026] For each transaction across all lake warehouses, and for each data item in the read set of that transaction, based on the read-after-write and write-after-read dependencies between transactions, the first transaction that writes the data item closest to the time of the current transaction is selected from all cross-lake warehouse transactions. It is then determined whether the first transaction has been removed. If the first transaction has been removed, no action is taken. If the first transaction has not been removed, a directed edge is constructed in the initial dependency graph between the node corresponding to the current transaction and the node corresponding to the first transaction, pointing from the node of the current transaction to the node of the first transaction. After the above processing, the first dependency graph is obtained for all data items of all transactions.

[0027] Optionally, step 2.3.1.3 specifically includes:

[0028] For each transaction across all lake warehouses and each data item in its write set, based on read-after-write and write-after-read dependencies between transactions, among all transactions across lake warehouses, find the second transaction that writes the data item with the closest time to the current transaction. Determine if the second transaction has been removed. If the second transaction has been removed, no action is taken. If the second transaction has not been removed, in the first dependency graph, construct a directed edge from the node of the first transaction to the node of the second transaction. After the above processing, all data items of all transactions obtain the second dependency graph. Next, find the transactions that read the data item between the times of the first and second transactions across all lake warehouses, forming a reader set. For each third transaction in the reader set, determine if the third transaction has been removed. If the third transaction has been removed, no action is taken. If the third transaction has not been removed, in the second dependency graph, construct a directed edge from the node of the first transaction to the node of the third transaction. After the above processing, all data items of all transactions obtain the local dependency graph.

[0029] Optionally, step 2.3.3 specifically includes:

[0030] At the two edges of the current time period, the sequences representing partial order relationships in the current lake warehouse server are sent to other lake warehouse servers, and the sequences representing partial order relationships sent by other lake warehouse servers are received, resulting in sequences representing partial order relationships for all lake warehouse servers. For each sequence representing a partial order relationship, a partial order relationship dependency graph is constructed. The partial order relationship dependency graph contains multiple nodes, each node corresponding to a transaction of all lake warehouse servers. There are directed edges between nodes, and the directed edges represent the partial order relationships between nodes, thus obtaining the partial order relationship dependency graph of all lake warehouse servers. The partial order relationship dependency graphs of all lake warehouse servers are merged to obtain a global relationship dependency graph. Among all transactions in the global relationship dependency graph, the transaction to be removed is determined, and the node and directed edge corresponding to the removed transaction are removed from the global relationship dependency graph, resulting in a target relationship dependency graph. Based on the Tarjan algorithm, strongly connected components (SCCs) in the target relationship dependency graph are found, and based on all strongly connected components (SCCs), the transactions in the target relationship dependency graph are sorted using a topological sorting algorithm to generate a global order.

[0031] Optionally, the removal transaction forms a circular dependency with other transactions, and the dependency relationship between the removal transaction and other transactions in the global dependency graph is less than the dependency relationship between other transactions and other transactions in the global dependency graph; the removal transaction retains its unique identifier TID and is added to the transactions in the next time period.

[0032] The beneficial effects of adopting the above technical solution are as follows:

[0033] Compared to traditional consistency commit strategies based on two-phase commit protocols and consensus protocols, this invention significantly reduces communication rounds in the context of wide area network (WAN) communication. Two-phase commit requires two WAN round trips to ensure transaction consistency, while consensus protocols like Paxos require at least two WAN round trips to coordinate distributed transaction commits and maintain consistent replicas. The concurrency control algorithm proposed in this invention utilizes the principle of data locality. Transactions within a local lakehouse do not require WAN round trips, while cross-lakehouse transactions only require one WAN round trip, reducing overall communication latency by one round. Simultaneously, transaction segmentation based on time intervals amortizes transaction latency across transactions within an epoch, reducing average transaction latency. The optimistic asynchronous execution and speculative read strategies improve data visibility within the same epoch, especially since transactions in a lakehouse architecture are typically long transactions, providing performance improvements for transaction execution within that architecture. Attached Figure Description

[0034] Figure 1This is a flowchart illustrating a cross-domain transaction concurrency control method based on a lakeware fusion architecture, as described in an embodiment of the present invention. Detailed Implementation

[0035] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0036] To address the problems existing in the prior art, this invention provides a cross-domain transaction concurrency control method based on a lakeware fusion architecture, combined with... Figure 1 This may include the following steps:

[0037] Step 1: Send each transaction to a lake warehouse server. Each transaction contains multiple operations and has a corresponding read set, write set, and parameters. Based on the local clock, assign a globally unique identifier (TID) to each transaction and write the read set, write set, and parameters of the transaction to the lake warehouse server corresponding to the transaction. Obtain the underlying lake warehouse snapshot at the time corresponding to the globally unique identifier (TID). The underlying lake warehouse snapshot includes the time corresponding to the globally unique identifier (TID) and the status of the data in the lake warehouse server.

[0038] Step 2: Sort all transactions on all lake warehouse servers for the current time period to obtain the global order;

[0039] Step 2.1: For the current lake warehouse server, obtain all transactions within the current time period, that is, all transactions within the epoch. There are read-after-write dependencies and write-after-read dependencies between transactions.

[0040] Step 2.2: Determine whether the transaction crosses a lake warehouse. If the transaction does not cross a lake warehouse, execute the transaction directly, saving subsequent WAN communication. If the transaction crosses a lake warehouse, obtain all transactions that cross lake warehouses and execute Step 2.3.

[0041] Step 2.3: Sort all transactions across lake warehouses to obtain a sequence representing the partial order relationship. Based on the sequence representing the partial order relationship, generate a global order.

[0042] Step 2.3.1: Generate a local dependency graph based on the read-after-write and write-after-read dependencies that exist between transactions;

[0043] Step 2.3.1.1: Construct an initial dependency graph, which contains multiple nodes, each node corresponding to one transaction across all lake warehouse transactions;

[0044] Step 2.3.1.2: For each transaction in all cross-lake warehouse transactions, and for each data item in the readset of that transaction, update the initial dependency graph to obtain the first dependency graph based on the read-after-write dependency and write-after-read dependency relationships between transactions;

[0045] Specifically, for each transaction T in all cross-lake warehouse transactions, and each data item D in the read set of that transaction, based on the read-after-write dependency and write-after-read dependency relationships between transactions, among all cross-lake warehouse transactions, the first transaction T* that writes to data item D closest to the time of transaction T is obtained. It is then determined whether the first transaction T* has been removed. If the first transaction T* has been removed, no processing is performed. If the first transaction T* has not been removed, i.e., T*.removed is false, in the initial dependency graph, a directed edge is constructed between the node corresponding to the transaction and the node corresponding to the first transaction, pointing from the node of transaction T to the node of the first transaction T*. This directed edge indicates that the read operation of T must be performed after the write operation of T*. After the above processing, the first dependency graph is obtained for all data items of all transactions.

[0046] Step 2.3.1.3: For each transaction in all cross-lake warehouse transactions, and for each data item in the writeset of that transaction, update the first dependency graph to obtain the second dependency graph based on the read-after-write dependency and write-after-read dependency relationships between transactions, and further update the second dependency graph to obtain the local dependency graph.

[0047] Specifically, for each transaction T across all lake warehouse transactions, and for each data item D in the write set of that transaction, based on the read-after-write and write-after-read dependencies between transactions, among all lake warehouse transactions, the second transaction T** that writes to data item D most closely related to transaction T is identified. It is then determined whether the second transaction T** has been removed. If the second transaction T** has been removed, no action is taken. If the second transaction T** has not been removed (i.e., T**.removed is false), a directed edge is constructed in the first dependency graph between the node corresponding to transaction T and the node corresponding to the second transaction T**, pointing from the node of transaction T to the node of the second transaction T**. All data items of all transactions pass through this edge. After the above processing, a second dependency graph is obtained. Between the time corresponding to transaction T and the time corresponding to the second transaction T**, all transactions across the lake warehouse that read this data item form a reader set. For each third transaction T*** in the reader set, it is determined whether the third transaction T*** has been removed. If the third transaction T*** has been removed, no processing is performed. If the third transaction T*** has not been removed (i.e., T***.removed is false), a directed edge is constructed between the node corresponding to transaction T and the node corresponding to the third transaction T*** in the second dependency graph, pointing from the node of transaction T to the node of the third transaction T***. After the above processing, a local dependency graph is obtained for all data items of all transactions.

[0048] Step 2.3.2: Based on the topological sorting algorithm, generate a position for each transaction in the local dependency graph to obtain a sequence representing the partial order relationship;

[0049] Among them, the partial order relation refers to the order in which transactions are executed. For example, transaction B depends on transaction A, and transaction C depends on neither transaction A nor transaction B. The partial order relation of transactions A, B and C can be that transactions A and C are executed simultaneously, and transaction B is executed after they are completed.

[0050] In practice, a partial sorter generates a locally ordered sequence of transactions based on the read and write sets of the transactions. Specifically, the partial sorter generates a position for each transaction and ensures that the local sequence remains consistent across all other generated partial order relations, thus obtaining a sequence representing the partial order relation.

[0051] Step 2.3.3: Generate a global order based on the sequences representing partial order relations in all lake warehouse servers.

[0052] Specifically, at the two edges of the current time period, the sequences representing partial order relationships in the current lake warehouse server are sent to other lake warehouse servers, and sequences representing partial order relationships sent by other lake warehouse servers are received, resulting in sequences representing partial order relationships for all lake warehouse servers. For each sequence representing a partial order relationship, a partial order relationship dependency graph is constructed. The partial order relationship dependency graph contains multiple nodes, each node corresponding to a transaction of all lake warehouse servers. There are directed edges between nodes, and the directed edges represent the partial order relationships between nodes, thus obtaining the partial order relationship dependency graph of all lake warehouse servers. The partial order relationship dependency graphs of all lake warehouse servers are merged to obtain a global relationship dependency graph. Among all transactions in the global relationship dependency graph, the transaction to be removed is determined, and the node and directed edge corresponding to the removed transaction are removed from the global relationship dependency graph, resulting in a target relationship dependency graph. Based on the Tarjan algorithm, strongly connected components (SCCs) in the target relationship dependency graph are found, and based on all strongly connected components (SCCs), the transactions in the target relationship dependency graph are sorted using a topological sorting algorithm to generate a global order.

[0053] The removal transaction forms a circular dependency with other transactions, and the dependency relationship between the removal transaction and other transactions in the global relationship dependency graph is less than the dependency relationship between other transactions and other transactions in the global relationship dependency graph. The removal transaction retains its unique identifier TID and is added to the transaction in the next time period.

[0054] Circular dependency refers to the cyclic relationship between multiple transactions, that is, the dependency relationship between multiple transactions forms a closed loop. For example, transaction B depends on transaction A, transaction C depends on transaction B, and transaction A depends on transaction C. At this time, the dependency relationship between transaction A, transaction B, and transaction C forms a closed loop. For each of transaction A, transaction B, and transaction C, determine the number of their dependencies in the global dependency graph, and select the transaction with fewer dependencies as the transaction to be removed.

[0055] The step of determining which transaction to remove and removing the corresponding node and directed edge from the global dependency graph is to ensure the existence of topological sorting.

[0056] Because a speculative read strategy is used to improve data visibility, the read of a visible version may not be successfully committed at the edge of the epoch. It is necessary to re-verify whether the currently read visible version has become a committed version before committing. Otherwise, the transaction should be rolled back, which is implemented through step 3.

[0057] Step 3: For each transaction in the global order, determine whether there are conflicts among all operations within the transaction and between transactions. When there are no conflicts among all operations within the transaction and between transactions, execute all transactions in the global order according to the global order and the underlying lakehouse snapshot to obtain visible version data. When there are conflicts among all operations within the transaction and between transactions, execute the operations within the transaction serially according to the underlying lakehouse snapshot and execute all transactions in parallel to obtain visible version data. When there are conflicts among all operations within the transaction and between transactions, pass the transaction and retain its unique identifier TID, and add it to the transactions in the next time period.

[0058] Among them, the transaction that passes and retains its unique identifier TID and is added to the transaction in the next time period will have a better position, that is, a earlier position, when a position is generated for each transaction in step 2.3.2 of the next time period.

[0059] The visible version of the data after execution is available for subsequent transactions to read, improving data visibility within an epoch. During execution, due to the global execution order, transactions across lake warehouses avoid the overhead of synchronization and coordination, improving performance, especially in update-intensive transaction scenarios.

[0060] Step 4: In global order, determine whether the visible version data of the previous transaction of the current transaction can be converted into committed version data. If there are no conflicts in all operations within the transaction and no conflicts between transactions, or if there are conflicts in all operations within the transaction and no conflicts between transactions, it indicates that the visible version data of the previous transaction of the current transaction can be converted into committed version data. In global order, write the visible data of the transaction back to the snapshot. If there are conflicts in all operations within the transaction and conflicts between transactions, it indicates that the visible version data of the previous transaction of the current transaction cannot be converted into committed version data. In this case, take the previous transaction of the current transaction as the current transaction and return to execution: determine whether the visible version data of the previous transaction of the current transaction can be converted into committed version data. After the above processing, all transactions in the global order obtain the first snapshot table.

[0061] Step 5: Write the execution results of the transactions that do not cross lake warehouses in Step 2.2 to the global snapshot, and combine them with the first snapshot table obtained in Step 4 to generate a global snapshot table.

[0062] In this context, the consistency of the global snapshot is equivalent to the data consistency of transactions. To prevent the table from becoming too large, a deterministic replacement strategy and a method of tracking hot records are used to limit the size of the merged table.

[0063] Specifically, when there are no conflicts among all operations within a transaction and no conflicts between transactions, all transactions in the global order are executed according to the global order and the underlying lake warehouse snapshot, and the visible data of the transaction is written back to the snapshot, which is called commit. This can increase data visibility while reducing the average latency of transactions within an epoch.

[0064] Step 3 is equivalent to conflict detection before commit. Since the speculative read strategy is used to improve data visibility, the read of the visible version may not be successfully committed at the edge of the epoch. It is necessary to re-verify whether the currently read visible version has become the committed version before committing. Otherwise, the transaction should be rolled back.

[0065] The purpose of this invention is to propose a transaction concurrency control algorithm that improves data visibility and update efficiency while ensuring data consistency within a lakeware fusion architecture. To increase concurrency between transactions, partial order execution is used instead of total order relation to enhance the flexibility of transaction execution. Time interval-based transaction partitioning execution is used to distribute the latency across each transaction, while simultaneously increasing the data visibility of transactions within the time interval.

[0066] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A cross-domain transaction concurrency control method based on a lakeware fusion architecture, characterized in that, include: Step 1: Send each transaction to a lake warehouse server. Each transaction contains multiple operations and has a corresponding read set, write set, and parameters. Based on the local clock, assign a globally unique identifier (TID) to each transaction and write the read set, write set, and parameters of the transaction to the lake warehouse server corresponding to the transaction. Obtain the underlying lake warehouse snapshot at the time corresponding to the globally unique identifier (TID). The underlying lake warehouse snapshot includes the time corresponding to the globally unique identifier (TID) and the status of the data in the lake warehouse server. Step 2: Sort all transactions on all lake warehouse servers for the current time period to obtain the global order; Step 2.1: For the current lake warehouse server, obtain all transactions within the current time period. There are read-after-write dependencies and write-after-read dependencies between transactions; Step 2.2: Determine whether the transaction crosses a lake warehouse. If the transaction does not cross a lake warehouse, execute the transaction directly. If the transaction crosses a lake warehouse, obtain all transactions that cross lake warehouses and execute Step 2.

3. Step 2.3: Sort all transactions across lake warehouses to obtain a sequence representing the partial order relationship. Based on the sequence representing the partial order relationship, generate a global order. Step 3: For each transaction in the global sequence, determine whether there are any conflicts among all operations within the transaction, and whether there are any conflicts between transactions; When there are no conflicts among all operations within a transaction and no conflicts between transactions, all transactions in the global order are executed according to the global order and the underlying lakehouse snapshot to obtain visible version data; when there are conflicts among all operations within a transaction and no conflicts between transactions, the operations within the transaction are executed serially according to the underlying lakehouse snapshot, and all transactions are executed in parallel to obtain visible version data; when there are conflicts among all operations within a transaction and conflicts between transactions, the transaction is passed on and its unique identifier TID is retained, and it is added to the transactions of the next time period. Step 4: In global order, determine whether the visible version data of the previous transaction of the current transaction can be converted into the committed version data. If there are no conflicts in all operations within the transaction and no conflicts between transactions, or if there are conflicts in all operations within the transaction and no conflicts between transactions, it indicates that the visible version data of the previous transaction of the current transaction can be converted into the committed version data. In global order, write the visible data of the transaction back to the snapshot. When all operations within a transaction conflict and there are conflicts between transactions, if the visible version data of the previous transaction representing the current transaction cannot be converted into committed version data, then the previous transaction of the current transaction is taken as the current transaction, and execution is returned: determine whether the visible version data of the previous transaction of the current transaction can be converted into committed version data. After the above processing, all transactions in the global order obtain the first snapshot table. Step 5: Write the execution results of the transactions that do not cross lake warehouses in Step 2.2 to the global snapshot, and combine them with the first snapshot table obtained in Step 4 to generate a global snapshot table.

2. The cross-domain transaction concurrency control method based on a lakeware fusion architecture according to claim 1, characterized in that, Step 2.3 specifically includes: Step 2.3.1: Generate a local dependency graph based on the read-after-write and write-after-read dependencies that exist between transactions; Step 2.3.2: Based on the topological sorting algorithm, generate a position for each transaction in the local dependency graph to obtain a sequence representing the partial order relationship; Step 2.3.3: Generate a global order based on the sequences representing partial order relations in all lake warehouse servers.

3. The cross-domain transaction concurrency control method based on a lakeware fusion architecture according to claim 2, characterized in that, Step 2.3.1 specifically includes: Step 2.3.1.1: Construct an initial dependency graph, which contains multiple nodes, each node corresponding to one transaction across all lake warehouse transactions; Step 2.3.1.2: For each transaction in all cross-lake warehouse transactions, and for each data item in the read set of that transaction, update the initial dependency graph to obtain the first dependency graph based on the read-after-write dependency and write-after-read dependency relationships between transactions. Step 2.3.1.3: For each transaction in all cross-lake warehouse transactions, and for each data item in the write set of that transaction, update the first dependency graph to obtain the second dependency graph based on the read-after-write dependency and write-after-read dependency relationships between transactions, and further update the second dependency graph to obtain the local dependency graph.

4. The cross-domain transaction concurrency control method based on a lakeware fusion architecture according to claim 3, characterized in that, Step 2.3.1.2 specifically includes: For each transaction across all lake warehouses, and for each data item in the read set of that transaction, based on the read-after-write and write-after-read dependencies between transactions, the first transaction that writes the data item closest to the time of the current transaction is selected from all cross-lake warehouse transactions. It is then determined whether the first transaction has been removed. If the first transaction has been removed, no action is taken. If the first transaction has not been removed, a directed edge is constructed in the initial dependency graph between the node corresponding to the current transaction and the node corresponding to the first transaction, pointing from the node of the current transaction to the node of the first transaction. After the above processing, the first dependency graph is obtained for all data items of all transactions.

5. The cross-domain transaction concurrency control method based on a lakeware fusion architecture according to claim 3, characterized in that, Step 2.3.1.3 specifically includes: For each transaction across all lake warehouses and each data item in its write set, based on read-after-write and write-after-read dependencies between transactions, among all transactions across lake warehouses, find the second transaction that writes the data item with the closest time to the current transaction. Determine if the second transaction has been removed. If the second transaction has been removed, no action is taken. If the second transaction has not been removed, in the first dependency graph, construct a directed edge from the node of the first transaction to the node of the second transaction. After the above processing, all data items of all transactions obtain the second dependency graph. Next, find the transactions that read the data item between the times of the first and second transactions across all lake warehouses, forming a reader set. For each third transaction in the reader set, determine if the third transaction has been removed. If the third transaction has been removed, no action is taken. If the third transaction has not been removed, in the second dependency graph, construct a directed edge from the node of the first transaction to the node of the third transaction. After the above processing, all data items of all transactions obtain the local dependency graph.

6. The cross-domain transaction concurrency control method based on a lakeware fusion architecture according to claim 2, characterized in that, Step 2.3.3 specifically includes: At the two edges of the current time period, the sequences representing partial order relationships in the current lake warehouse server are sent to other lake warehouse servers, and the sequences representing partial order relationships sent by other lake warehouse servers are received, resulting in sequences representing partial order relationships for all lake warehouse servers. For each sequence representing a partial order relationship, a partial order relationship dependency graph is constructed. The partial order relationship dependency graph contains multiple nodes, each node corresponding to a transaction of all lake warehouse servers. There are directed edges between nodes, and the directed edges represent the partial order relationships between nodes, thus obtaining the partial order relationship dependency graph of all lake warehouse servers. The partial order relationship dependency graphs of all lake warehouse servers are merged to obtain a global relationship dependency graph. Among all transactions in the global relationship dependency graph, the transaction to be removed is determined, and the node and directed edge corresponding to the removed transaction are removed from the global relationship dependency graph, resulting in a target relationship dependency graph. Based on the Tarjan algorithm, strongly connected components (SCCs) in the target relationship dependency graph are found, and based on all strongly connected components (SCCs), the transactions in the target relationship dependency graph are sorted using a topological sorting algorithm to generate a global order.

7. The cross-domain transaction concurrency control method based on a lakeware fusion architecture according to claim 6, characterized in that, The removal transaction forms a circular dependency with other transactions, and the dependency relationship between the removal transaction and other transactions in the global relationship dependency graph is less than the dependency relationship between other transactions and other transactions in the global relationship dependency graph. The removed transaction retains its unique identifier TID and is added to the next transaction in the time period.

Images