Cross-domain transaction concurrency control method based on lake and warehouse fusion architecture

Abstract

The invention provides a cross-domain transaction concurrency control method based on a lake and warehouse fusion architecture, and relates to the technical field of transaction concurrency control. Each transaction is sent to a lake and warehouse server, and a global unique mark TID is allocated to each transaction based on a local clock; according to the cross-domain transaction concurrency control method based on the lake and warehouse fusion architecture, all transactions of all lake and warehouse servers in the current time period are sorted to obtain a global sequence, then conflict detection and transaction execution are carried out, and then submission verification is carried out to obtain a global snapshot table. Local lake and warehouse transactions do not need wide area network round-trip communication, cross-lake and warehouse transactions only need one round of wide area network round-trip communication, and the overall communication delay is reduced by one round. And meanwhile, the transaction division execution based on the time interval also enables the delay of the transaction to be shared by each transaction within one epoch, so that the average delay of the transaction is reduced. And the optimistic asynchronous execution and speculation reading strategies are used, so that the data visibility in the same epoch is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of transaction concurrency control, and specifically to a cross-domain transaction concurrency control method based on a lake-warehouse fusion architecture. Background Art

[0002] Transaction processing under the lake-warehouse fusion architecture is a key technology to ensure data consistency and integrity. A transaction refers to a series of operations performed as a single logical unit. For example, when performing real-time sales data analysis on an e-commerce platform under the lake-warehouse fusion architecture, at least the following operations on the data lake and data warehouse are included: writing every minute sales record to the log file in the data lake in real time; cleaning sales records to remove invalid data (such as duplicate records or format errors); aggregating sales data by hour, generating an "hourly sales" analysis table and saving it to the data warehouse; updating the metadata table and marking the "hourly sales" analysis table for query. Under normal circumstances, the above operations proceed smoothly, and a complete real-time data analysis task will eventually be completed, and all data status and metadata related to the task will be successfully updated. However, if a problem occurs in a certain link (for example, the log file is not fully written or the metadata table update fails), the entire operation will fail. Once a failure occurs, the system must ensure that the status of all data and metadata remains unchanged from the state before submission, and the transaction under the lake-warehouse fusion architecture is a technology used to ensure the stability and predictability of operations in this case.

[0003] Polaris is the core distributed SQL engine of Azure Synapse Analytics, designed for large-scale data processing in cloud environments. Polaris' transaction processing process adopts a distributed architecture, covering multiple stages such as transaction initiation, read and write operations, concurrency control, and storage optimization to achieve data consistency and efficiency. The client first initiates a transaction request, and the SQL front end (FE) creates a root transaction and assigns a unique transaction ID. The read operation is based on snapshot isolation semantics, which is completed by capturing a snapshot of the database table (generated by the Manifest file of the committed transaction), and the SQL back end (BE) executes the query in the snapshot to generate the result. In the write operation, the system generates the corresponding Parquet data file or delete vector file according to the insert, update, or delete operation, and writes the change record to the transaction manifest file. The transaction commit phase ensures the serialization order of the transaction by acquiring the commit lock and detects potential conflicts during the verification process; if there is no conflict, all changes are committed, otherwise they are rolled back to the initial state. At the same time, Polaris effectively avoids problems such as dirty reads, non-repeatable reads, and phantom reads through optimistic concurrency control (OCC) and snapshot isolation semantics.

[0004] NoisePage is an in-memory database management system (DBMS) that supports high-performance transaction processing workloads and optimizes the data export process with external data analysis tools. NoisePage's transaction processing mechanism combines optimistic concurrency control (OCC) with multi-version concurrency control (MVCC). By assigning a timestamp pair (start and commit) to each transaction, it maintains the version chain to support concurrent operations while avoiding write-write conflicts to reduce the risk of rollback. When a transaction is committed, the system assigns a commit timestamp and updates the log manager to ensure that the transaction changes are persistent; rollbacks use undo records to restore data. Write-ahead logging and checkpoint mechanisms ensure persistence, and transactions are marked as committed after the log is written to disk. Garbage collection ensures efficient use of system resources by pruning version chains and freeing up memory to clean up invalid records, thereby achieving consistent and high-performance transaction processing.

[0005] The lake-warehouse fusion architecture uses object storage in an open data format. Existing methods all have the problem of high overhead of the consistency guarantee mechanism, and transaction concurrency control does not perform well on the lake-warehouse architecture.

[0006] Polaris uses SQLDB for transaction management, which increases the complexity of the system. At the same time, the multi-level Manifest file and WriteSets table structure increase the consistency requirements after the transaction is executed, requiring more synchronization coordination and verification. If the Polaris system is distributed across multiple data centers, consistency and transaction processing across data centers may become more complicated, and additional mechanisms are required to ensure data consistency in a distributed environment. In the case of high-concurrency writes, certain delays may occur due to the need to coordinate multiple concurrent write operations. At the same time, although the system is designed as a stateless computing architecture, in some extreme cases (such as large-scale node failures), error recovery and rollback processes may still face challenges.

[0007] NoisePage has a write amplification problem. Specifically, when updating a variable-length value, the entire value buffer needs to be copied and the offset array needs to be updated, which increases the overhead of write operations. In general, in order to improve the efficiency of data export, NoisePage uses the Arrow data format at the expense of transaction processing speed. The complex mechanism and specific data format not only lead to the system throughput in update-intensive transaction scenarios showing worse performance than existing methods. Summary of the invention

[0008] In view of the deficiencies of the prior art, the purpose of the present invention is to propose a cross-domain transaction concurrency control method based on the lake warehouse fusion architecture, including:

[0009] Step 1: Send each transaction to a lake warehouse server. The transaction contains multiple operations. The transaction corresponds to a read set, a write set and parameters. Based on the local clock, a globally unique mark TID is assigned to each transaction, and the read set, write set and parameters of the transaction are written to the lake warehouse server corresponding to the transaction. The underlying lake warehouse snapshot at the time corresponding to the globally unique mark TID is obtained. The underlying lake warehouse snapshot includes the time corresponding to the globally unique mark TID and the status of the data in the lake warehouse server;

[0010] Step 2: Sort all transactions in the current time period of all Hucang servers to obtain the global order;

[0011] Step 2.1: For the current Hucang server, obtain all transactions in the current time period. There are read-after-write dependencies and write-after-read dependencies between transactions;

[0012] Step 2.2: Determine whether the transaction crosses the lake warehouse. If the transaction does not cross the lake warehouse, execute the transaction directly; if the transaction crosses the lake warehouse, obtain all cross-lake warehouse transactions and execute step 2.3;

[0013] Step 2.3: Sort all cross-lake warehouse transactions to obtain a sequence representing a partial order relationship, and generate a global order based on the sequence representing the partial order relationship;

[0014] Step 3: For each transaction in the global order, determine whether there are conflicts among all operations in the transaction and among transactions; when there are no conflicts among all operations in the transaction and among transactions, execute all transactions in the global order according to the global order and the underlying lake warehouse snapshot to obtain visible version data; when there are conflicts among all operations in the transaction and among transactions, execute the operations in the transaction serially and all transactions in parallel according to the underlying lake warehouse snapshot to obtain visible version data; when there are conflicts among all operations in the transaction and among transactions, pass the transaction and retain its unique tag TID and add it to the transaction of the next time period;

[0015] Step 4: In global order, determine whether the visible version data of the previous transaction of the current transaction can be converted into the committed version data. When all operations within the transaction do not conflict and there are no conflicts between transactions, or when all operations within the transaction do conflict and there are no conflicts between transactions, the visible version data of the previous transaction representing the current transaction can be converted into the committed version data. In global order, write the visible data of the transaction back to the snapshot. When all operations within the transaction conflict and there are conflicts between transactions, the visible version data of the previous transaction representing the current transaction cannot be converted into the committed version data. Then, take the previous transaction of the current transaction as the current transaction and return to execute: determine whether the visible version data of the previous transaction of the current transaction can be converted into the committed version data. After all transactions in the global order are processed as above, the first snapshot table is obtained.

[0016] Step 5: Write the execution results of the transaction that does not cross the lake warehouse in step 2.2 into the global snapshot, and combine it with the first snapshot table obtained in step 4 to generate a global snapshot table.

[0017] Optionally, step 2.3 specifically includes:

[0018] Step 2.3.1: Generate a local dependency graph based on the read-after-write dependency and write-after-read dependency between transactions;

[0019] Step 2.3.2: Based on the topological sorting algorithm, generate a position for each transaction in the local dependency graph to obtain a sequence representing the partial order relationship;

[0020] Step 2.3.3: Generate a global order based on the sequences representing the partial order relationship in all lake warehouse servers.

[0021] Optionally, step 2.3.1 specifically includes:

[0022] Step 2.3.1.1: Build an initial dependency graph, where the initial dependency graph contains multiple nodes, each of which corresponds to a transaction among all transactions across the lake warehouse;

[0023] Step 2.3.1.2: For each transaction in all cross-lake warehouse transactions and each data item in the read set of the transaction, according to the existence of read-after-write dependency and write-after-read dependency between transactions, the initial dependency graph is updated to obtain the first dependency graph;

[0024] Step 2.3.1.3: For each transaction in all cross-lake warehouse transactions and each data item in the write set of the transaction, based on the existence of read-after-write dependency and write-after-read dependency between transactions, the first dependency graph is updated to obtain the second dependency graph, and the second dependency graph is further updated to obtain the local dependency graph.

[0025] Optionally, step 2.3.1.2 specifically includes:

[0026] For each transaction among all transactions across the lake warehouse, and each data item in the read set of the transaction, according to the read-after-write dependency and write-after-read dependency between transactions, among all transactions across the lake warehouse, obtain a first transaction that writes the data item closest in time to the transaction, and determine whether the first transaction is removed. If the first transaction is removed, do not do anything. If the first transaction is not removed, in the initial dependency graph, between the node corresponding to the transaction and the node corresponding to the first transaction, construct a directed edge from the node of the transaction to the node of the first transaction. After the above processing, all data items of all transactions obtain the first dependency graph.

[0027] Optionally, step 2.3.1.3 specifically includes:

[0028] For each transaction in all transactions across the lake warehouse, and for each data item in the write set of the transaction, according to the existence of read-after-write dependency and write-after-read dependency between transactions, among all transactions across the lake warehouse, obtain a second transaction that writes the data item closest to the time of the transaction, and determine whether the second transaction is removed. If the second transaction is removed, no processing is performed. If the second transaction is not removed, in the first dependency graph, between the node corresponding to the transaction and the node corresponding to the second transaction, a directed edge from the node of the transaction to the node of the second transaction is constructed. After the above processing, all data items of all transactions are processed to obtain the second dependency graph; between the time corresponding to the transaction and the time corresponding to the second transaction, the transactions that read the data item in all transactions across the lake warehouse are obtained to form a reader set. For each third transaction in the reader set, determine whether the third transaction is removed. If the third transaction is removed, no processing is performed. If the third transaction is not removed, in the second dependency graph, between the node corresponding to the transaction and the node corresponding to the third transaction, a directed edge from the node of the transaction to the node of the third transaction is constructed. After the above processing, all data items of all transactions are processed to obtain a local dependency graph.

[0029] Optionally, step 2.3.3 specifically includes:

[0030] At the two edges of the current time period, the sequence representing the partial order relationship in the current lake warehouse server is sent to other lake warehouse servers, and the sequences representing the partial order relationship sent by other lake warehouse servers are received to obtain the sequences representing the partial order relationship of all lake warehouse servers. For each sequence representing the partial order relationship, a partial order relationship dependency graph is constructed, and the partial order relationship dependency graph includes multiple nodes, each node corresponds to a transaction of all lake warehouse servers, and there are directed edges between the nodes. The directed edges represent the partial order relationship between the nodes, and then the partial order relationship dependency graph of all lake warehouse servers is obtained. The partial order relationship dependency graphs of all lake warehouse servers are merged to obtain a global relationship dependency graph. Among all transactions in the global relationship dependency graph, the removed transaction is determined, and the nodes and directed edges corresponding to the removed transaction are removed from the global relationship dependency graph to obtain the target relationship dependency graph. Based on the Tarjan algorithm, the strongly connected components SCCs in the target relationship dependency graph are found, and according to all the strongly connected components SCCs, the transactions in the target relationship dependency graph are sorted by a topological sorting algorithm to generate a global order.

[0031] Optionally, a circular dependency is formed between the removed transaction and other transactions, and the dependency relationship between the removed transaction and transactions other than the removed transaction and other transactions in the global relationship dependency graph is less than the dependency relationship between other transactions and transactions other than the removed transaction and other transactions in the global relationship dependency graph; the removed transaction retains its unique identifier TID and is added to the transactions in the next time period.

[0032] The beneficial effects of adopting the above technical solution are:

[0033] Compared with the traditional consistency commit strategy based on the two-phase commit protocol and the consensus protocol, in terms of communication saving, the communication rounds are significantly reduced in the context of wide area network communication. The two-phase commit requires two rounds of wide area network round-trip communication to ensure the consistency of transaction submission, while the use of consensus protocols such as Paxos requires at least two rounds of wide area network round-trip communication to coordinate the submission of distributed transactions and the maintenance of consistent copies. The concurrency control algorithm proposed in the present invention utilizes the principle of data locality. Transactions in local lake warehouses do not require wide area network round-trip communication, while transactions across lake warehouses only require one round of wide area network round-trip communication, and the overall communication delay is reduced by one round. At the same time, the transaction segmentation execution based on time intervals also allows the transaction delay to be amortized by each transaction within an epoch, reducing the average transaction delay. The optimistic asynchronous execution and speculative read strategies used improve the data visibility within the same epoch, especially since the transactions under the lake warehouse architecture are all long transactions, which provides performance improvement for transaction execution under the lake warehouse architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1The present invention is a flowchart of a cross-domain transaction concurrency control method based on a lake-warehouse fusion architecture in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0036] In view of the problems existing in the prior art, the present invention provides a cross-domain transaction concurrency control method based on the lake warehouse fusion architecture. Figure 1 , which may include the following steps:

[0037] Step 1: Send each transaction to a lake warehouse server. The transaction contains multiple operations. The transaction corresponds to a read set, a write set and parameters. Based on the local clock, a globally unique mark TID is assigned to each transaction, and the read set, write set and parameters of the transaction are written to the lake warehouse server corresponding to the transaction. The underlying lake warehouse snapshot at the time corresponding to the globally unique mark TID is obtained. The underlying lake warehouse snapshot includes the time corresponding to the globally unique mark TID and the status of the data in the lake warehouse server;

[0038] Step 2: Sort all transactions in the current time period of all Hucang servers to obtain the global order;

[0039] Step 2.1: For the current Hucang server, obtain all transactions in the current time period, that is, all transactions in the epoch. There are read-after-write dependencies and write-after-read dependencies between transactions;

[0040] Step 2.2: Determine whether the transaction crosses the lake warehouse. If the transaction does not cross the lake warehouse, directly execute the transaction, eliminating the need for subsequent WAN communication. If the transaction crosses the lake warehouse, obtain all cross-lake warehouse transactions and execute step 2.3.

[0041] Step 2.3: Sort all cross-lake warehouse transactions to obtain a sequence representing a partial order relationship, and generate a global order based on the sequence representing the partial order relationship;

[0042] Step 2.3.1: Generate a local dependency graph based on the read-after-write dependency and write-after-read dependency between transactions;

[0043] Step 2.3.1.1: Build an initial dependency graph, where the initial dependency graph contains multiple nodes, each of which corresponds to a transaction among all transactions across the lake warehouse;

[0044] Step 2.3.1.2: For each transaction in all cross-lake warehouse transactions and each data item in the read set of the transaction, according to the existence of read-after-write dependency and write-after-read dependency between transactions, the initial dependency graph is updated to obtain the first dependency graph;

[0045] Specifically, for each transaction T in all transactions across the lake warehouse, and each data item D in the read set of the transaction, according to the existence of read-after-write dependency and write-after-read dependency between transactions, a first transaction T* that writes the data item D closest in time to the transaction T is obtained in all transactions across the lake warehouse, and it is determined whether the first transaction T* is removed. If the first transaction T* is removed, no processing is performed. If the first transaction T* is not removed, that is, T*.removed is false, in the initial dependency graph, between the node corresponding to the transaction and the node corresponding to the first transaction, a directed edge is constructed from the node of the transaction T to the node of the first transaction T*. This directed edge indicates that the read operation of T must be performed after the write operation of T*. After the above processing, all data items of all transactions obtain the first dependency graph.

[0046] Step 2.3.1.3: For each transaction in all cross-lake warehouse transactions, and each data item in the write set of the transaction, according to the existence of read-after-write dependency and write-after-read dependency between transactions, the first dependency graph is updated to obtain the second dependency graph, and the second dependency graph is further updated to obtain the local dependency graph.

[0047] Specifically, for each transaction T in all transactions across the lake warehouse, and each data item D in the write set of the transaction, according to the read-after-write dependency and write-after-read dependency between transactions, obtain a second transaction T** that writes the data item D closest to the time of the transaction T in all transactions across the lake warehouse, and determine whether the second transaction T** has been removed. If the second transaction T** has been removed, no processing is performed. If the second transaction T** has not been removed, that is, T**.removed is false, in the first dependency graph, between the node corresponding to the transaction T and the node corresponding to the second transaction T**, a directed edge is constructed from the node of the transaction T to the node of the second transaction T**, and all data items of all transactions pass through After the above processing, the second dependency graph is obtained; the transactions that read the data item in all cross-lake transactions between the time corresponding to the transaction T and the time corresponding to the second transaction T** are obtained to form a reader set. For each third transaction T*** in the reader set, it is determined whether the third transaction T*** is removed. If the third transaction T*** is removed, no processing is performed. If the third transaction T*** is not removed, that is, T***.removed is false, in the second dependency graph, between the node corresponding to the transaction T and the node corresponding to the third transaction T***, a directed edge is constructed from the node of the transaction T to the node of the third transaction T***. After the above processing, all data items of all transactions are processed to obtain a local dependency graph.

[0048] Step 2.3.2: Based on the topological sorting algorithm, generate a position for each transaction in the local dependency graph to obtain a sequence representing the partial order relationship;

[0049] Among them, the partial order relationship refers to the sequential relationship of transaction execution. For example, transaction B depends on transaction A, and transaction C depends on neither transaction A nor transaction B. The partial order relationship of transaction A, transaction B and transaction C can be that transaction A and transaction C are executed at the same time, and transaction B is executed after the execution is completed.

[0050] In the specific implementation process, the partial sorter generates a locally ordered transaction sequence based on the read-write set of the transaction. Specifically, the partial sorter generates a position for each transaction and ensures that the local sequence of the position is consistent in all other generated partial partial order relations, thereby obtaining a sequence that represents the partial order relation.

[0051] Step 2.3.3: Generate a global order based on the sequences representing the partial order relationship in all lake warehouse servers.

[0052] Specifically, at the two edges of the current time period, the sequence representing the partial order relationship in the current lake warehouse server is sent to other lake warehouse servers, and the sequences representing the partial order relationship sent by other lake warehouse servers are received to obtain the sequences representing the partial order relationship of all lake warehouse servers. For each sequence representing the partial order relationship, a partial order relationship dependency graph is constructed, and the partial order relationship dependency graph includes multiple nodes, each node corresponds to a transaction of all lake warehouse servers, and there are directed edges between the nodes, and the directed edges represent the partial order relationship between the nodes, thereby obtaining the partial order relationship dependency graph of all lake warehouse servers, merging the partial order relationship dependency graphs of all lake warehouse servers to obtain a global relationship dependency graph, determining to remove transactions from all transactions in the global relationship dependency graph, and removing the nodes and directed edges corresponding to the removed transactions from the global relationship dependency graph to obtain the target relationship dependency graph, and based on the Tarjan algorithm, finding the strongly connected components SCCs in the target relationship dependency graph, and sorting the transactions in the target relationship dependency graph according to all the strongly connected components SCCs through a topological sorting algorithm to generate a global order.

[0053] A circular dependency is formed between the removal transaction and other transactions, and the dependency relationship between the removal transaction and transactions other than the removal transaction and other transactions in the global relationship dependency graph is less than the dependency relationship between other transactions and transactions other than the removal transaction and other transactions in the global relationship dependency graph; the removal transaction retains its unique identifier TID and is added to the transactions of the next time period.

[0054] Among them, circular dependency means that the dependency relationship between multiple transactions is cyclic, that is, the dependency relationship between multiple transactions forms a closed loop. For example, transaction B depends on transaction A, transaction C depends on transaction B, and transaction A depends on transaction C. At this time, the dependency relationship between transaction A, transaction B and transaction C forms a closed loop. For transaction A, transaction B and transaction C respectively, determine the number of dependencies in the global dependency graph, and select transactions with fewer dependencies as removed transactions.

[0055] The purpose of determining to remove the transaction and removing the nodes and directed edges corresponding to the removed transaction from the global relationship dependency graph is to ensure the existence of topological sorting.

[0056] Since a speculative read strategy is used to improve data visibility, the reading of the visible version may not be successfully committed at the edge of the epoch. It is necessary to re-verify whether the currently read visible version has become the committed version before committing. Otherwise, the transaction should be rolled back, which is specifically achieved through step 3.

[0057] Step 3: For each transaction in the global order, determine whether there are conflicts among all operations in the transaction and among transactions; when there are no conflicts among all operations in the transaction and among transactions, execute all transactions in the global order according to the global order and the underlying lake warehouse snapshot to obtain visible version data; when there are conflicts among all operations in the transaction and among transactions, execute the operations in the transaction serially and all transactions in parallel according to the underlying lake warehouse snapshot to obtain visible version data; when there are conflicts among all operations in the transaction and among transactions, pass the transaction and retain its unique tag TID and add it to the transaction of the next time period;

[0058] Among them, the transaction that passes and retains its unique tag TID and is added to the transactions of the next time period has a better position, that is, a more forward position, when a position is generated for each transaction in step 2.3.2 of the next time period.

[0059] The visible version data after execution can be read by subsequent transactions, which improves the visibility of data within the epoch. During execution, due to the global execution order, transactions across lake warehouses are free from the overhead of synchronization and coordination, which improves the performance of update-intensive transaction scenarios.

[0060] Step 4: In global order, determine whether the visible version data of the previous transaction of the current transaction can be converted into the committed version data. When there is no conflict among all operations within the transaction and there is no conflict between transactions, or there is a conflict among all operations within the transaction and there is no conflict between transactions, the visible version data representing the previous transaction of the current transaction can be converted into the committed version data. In global order, write the visible data of the transaction back to the snapshot. When there is a conflict among all operations within the transaction and there is a conflict between transactions, the visible version data representing the previous transaction of the current transaction cannot be converted into the committed version data. Then, take the previous transaction of the current transaction as the current transaction and return to execute: determine whether the visible version data of the previous transaction of the current transaction can be converted into the committed version data. After the above processing, all transactions in the global order obtain the first snapshot table.

[0061] Step 5: Write the execution results of the transaction that does not cross the lake warehouse in step 2.2 into the global snapshot, and combine it with the first snapshot table obtained in step 4 to generate a global snapshot table.

[0062] The consistency of the global snapshot is equivalent to the data consistency of the transaction. To prevent the table from becoming too large, the size of the fusion table is limited by a deterministic replacement strategy and tracking of hot records.

[0063] Among them, when there is no conflict in all operations within the transaction and there is no conflict between transactions, all transactions in the global order are executed according to the global order and the underlying lake warehouse snapshot, and the visible data of the transaction is written back to the snapshot, that is, committed, which can increase data visibility while reducing the average latency of transactions within the epoch.

[0064] Among them, step 3 is equivalent to conflict detection before submission. Since the speculative read strategy is used to improve data visibility, the reading of the visible version may not be successfully submitted at the edge of the epoch. It is necessary to re-verify whether the currently read visible version has become the submitted version before submission, otherwise the transaction should be rolled back.

[0065] The purpose of this invention is to propose a transaction concurrency control algorithm that improves data visibility and update efficiency under the condition of ensuring data consistency under the lake warehouse fusion architecture. In order to increase the concurrency between transactions, partial order execution is used instead of full order relationship to improve the flexibility of transaction execution. The transaction split execution based on time interval is used to amortize the delay of each transaction, while increasing the data visibility of the transaction within the time interval.

[0066] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. A cross-domain transaction concurrency control method based on lake-warehouse fusion architecture, characterized in that: include: Step 1: Send each transaction to a lake warehouse server. The transaction contains multiple operations. The transaction corresponds to a read set, a write set and parameters. Based on the local clock, a globally unique mark TID is assigned to each transaction, and the read set, write set and parameters of the transaction are written to the lake warehouse server corresponding to the transaction. The underlying lake warehouse snapshot at the time corresponding to the globally unique mark TID is obtained. The underlying lake warehouse snapshot includes the time corresponding to the globally unique mark TID and the status of the data in the lake warehouse server; Step 2: Sort all transactions in the current time period of all Hucang servers to obtain the global order; Step 2.1: For the current Hucang server, obtain all transactions in the current time period. There are read-after-write dependencies and write-after-read dependencies between transactions; Step 2.2: Determine whether the transaction crosses the lake warehouse. If the transaction does not cross the lake warehouse, execute the transaction directly; if the transaction crosses the lake warehouse, obtain all cross-lake warehouse transactions and execute step 2.3; Step 2.3: Sort all cross-lake warehouse transactions to obtain a sequence representing a partial order relationship, and generate a global order based on the sequence representing the partial order relationship; Step 3: For each transaction in the global order, determine whether there are conflicts among all operations within the transaction, and whether there are conflicts between transactions; When all operations within a transaction do not conflict and there are no conflicts between transactions, all transactions in the global order are executed according to the global order and the underlying lake warehouse snapshot to obtain visible version data; when all operations within a transaction conflict and there are no conflicts between transactions, the operations within the transaction are executed serially and all transactions are executed in parallel according to the underlying lake warehouse snapshot to obtain visible version data; when all operations within a transaction conflict and there are conflicts between transactions, the transaction is passed and its unique tag TID is retained and added to the transaction of the next time period; Step 4: In global order, determine whether the visible version data of the previous transaction of the current transaction can be converted to the committed version data. If all operations within the transaction do not conflict and there are no conflicts between transactions, or if all operations within the transaction do conflict and there are no conflicts between transactions, the visible version data of the previous transaction representing the current transaction can be converted to the committed version data. In global order, write the visible data of the transaction back to the snapshot; When all operations in a transaction conflict and conflicts exist between transactions, and the visible version data representing the previous transaction of the current transaction cannot be converted into the committed version data, the previous transaction of the current transaction is taken as the current transaction, and the execution is returned to: determine whether the visible version data of the previous transaction of the current transaction can be converted into the committed version data. After all transactions in the global order are processed as above, a first snapshot table is obtained; Step 5: Write the execution results of the transaction that does not cross the lake warehouse in step 2.2 into the global snapshot, and combine it with the first snapshot table obtained in step 4 to generate a global snapshot table.

2. According to claim 1, a cross-domain transaction concurrency control method based on lake warehouse fusion architecture is characterized in that: Step 2.3 specifically includes: Step 2.3.1: Generate a local dependency graph based on the read-after-write dependency and write-after-read dependency between transactions; Step 2.3.2: Based on the topological sorting algorithm, generate a position for each transaction in the local dependency graph to obtain a sequence representing the partial order relationship; Step 2.3.3: Generate a global order based on the sequences representing the partial order relationship in all lake warehouse servers.

3. According to claim 2, a cross-domain transaction concurrency control method based on lake warehouse fusion architecture is characterized in that: Step 2.3.1 specifically includes: Step 2.3.1.1: Build an initial dependency graph, where the initial dependency graph contains multiple nodes, each of which corresponds to a transaction among all transactions across the lake warehouse; Step 2.3.1.2: For each transaction in all cross-lake warehouse transactions and each data item in the read set of the transaction, according to the existence of read-after-write dependency and write-after-read dependency between transactions, the initial dependency graph is updated to obtain the first dependency graph; Step 2.3.1.3: For each transaction in all cross-lake warehouse transactions and each data item in the write set of the transaction, based on the existence of read-after-write dependency and write-after-read dependency between transactions, the first dependency graph is updated to obtain the second dependency graph, and the second dependency graph is further updated to obtain the local dependency graph.

4. According to claim 3, a cross-domain transaction concurrency control method based on lake warehouse fusion architecture is characterized in that: Step 2.3.1.2 specifically includes: For each transaction among all transactions across the lake warehouse, and each data item in the read set of the transaction, according to the read-after-write dependency and write-after-read dependency between transactions, among all transactions across the lake warehouse, obtain a first transaction that writes the data item closest in time to the transaction, and determine whether the first transaction is removed. If the first transaction is removed, do not do anything. If the first transaction is not removed, in the initial dependency graph, between the node corresponding to the transaction and the node corresponding to the first transaction, construct a directed edge from the node of the transaction to the node of the first transaction. After the above processing, all data items of all transactions obtain the first dependency graph.

5. According to claim 3, a cross-domain transaction concurrency control method based on lake warehouse fusion architecture is characterized in that: Step 2.3.1.3 specifically includes: For each transaction in all transactions across the lake warehouse, and for each data item in the write set of the transaction, according to the existence of read-after-write dependency and write-after-read dependency between transactions, among all transactions across the lake warehouse, obtain a second transaction that writes the data item closest to the time of the transaction, and determine whether the second transaction is removed. If the second transaction is removed, no processing is performed. If the second transaction is not removed, in the first dependency graph, between the node corresponding to the transaction and the node corresponding to the second transaction, a directed edge from the node of the transaction to the node of the second transaction is constructed. After the above processing, all data items of all transactions are processed to obtain the second dependency graph; between the time corresponding to the transaction and the time corresponding to the second transaction, the transactions that read the data item in all transactions across the lake warehouse are obtained to form a reader set. For each third transaction in the reader set, determine whether the third transaction is removed. If the third transaction is removed, no processing is performed. If the third transaction is not removed, in the second dependency graph, between the node corresponding to the transaction and the node corresponding to the third transaction, a directed edge from the node of the transaction to the node of the third transaction is constructed. After the above processing, all data items of all transactions are processed to obtain a local dependency graph.

6. According to a method for controlling cross-domain transaction concurrency based on lake-warehouse fusion architecture according to claim 2, it is characterized in that: Step 2.3.3 specifically includes: At the two edges of the current time period, the sequence representing the partial order relationship in the current lake warehouse server is sent to other lake warehouse servers, and the sequences representing the partial order relationship sent by other lake warehouse servers are received to obtain the sequences representing the partial order relationship of all lake warehouse servers. For each sequence representing the partial order relationship, a partial order relationship dependency graph is constructed, and the partial order relationship dependency graph includes multiple nodes, each node corresponds to a transaction of all lake warehouse servers, and there are directed edges between the nodes. The directed edges represent the partial order relationship between the nodes, and then the partial order relationship dependency graph of all lake warehouse servers is obtained. The partial order relationship dependency graphs of all lake warehouse servers are merged to obtain a global relationship dependency graph. Among all transactions in the global relationship dependency graph, the removed transaction is determined, and the nodes and directed edges corresponding to the removed transaction are removed from the global relationship dependency graph to obtain the target relationship dependency graph. Based on the Tarjan algorithm, the strongly connected components SCCs in the target relationship dependency graph are found, and according to all the strongly connected components SCCs, the transactions in the target relationship dependency graph are sorted by a topological sorting algorithm to generate a global order.

7. According to claim 6, a cross-domain transaction concurrency control method based on lake warehouse fusion architecture is characterized in that: A circular dependency is formed between the removal transaction and other transactions, and the dependency relationship between the removal transaction and transactions other than the removal transaction and other transactions in the global relationship dependency graph is less than the dependency relationship between other transactions and transactions other than the removal transaction and other transactions in the global relationship dependency graph; The removed transaction retains its unique identifier TID and is added to the transactions of the next time period.

Images