Distributed concurrency control method and system based on fine-grained transaction repair

Abstract

The present invention provides a distributed concurrency control method and system based on fine-grained transaction repair, comprising: step S1: a computing node receives a user request and executes a transaction operation according to the request content; step S2: based on the result of executing the transaction operation, the computing node locally commits and queues the locally submitted transaction; step S3: when the computing node's local transaction queue is full, all transactions in the queue are packaged into a batch transaction, and the batch transaction is globally submitted to the storage node; if the batch transaction fails verification during the global submission process, the batch transaction is restored to a committable state through fine-grained transaction repair, and the batch transaction is continued to be submitted. The present invention utilizes the computing node's local cache to reduce the communication caused by the storage node data reading operation during transaction execution, thereby reducing transaction execution latency.

Description

Technical Field

[0001] The present invention relates to the fields of distributed transaction processing systems and distributed concurrency control, and in particular to a distributed concurrency control method and system based on fine-grained transaction repair. Background Art

[0002] In recent years, with the rapid development of cloud computing technology, there has been an increasing demand for migrating traditional distributed transaction processing systems to the cloud. These systems typically adopt a compute-storage architecture, where compute and storage logic are implemented within the same process or node. This design stems from the fact that system operators often also own the system hardware. Consequently, the compute and storage hardware resources on servers are typically fixed and coupled. Consequently, software design does not require the separation of compute and storage logic. However, the rise of cloud computing technology has enabled flexible configuration of hardware resources, placing new demands on the elasticity and scalability of software systems. Specifically, a system needs to be able to independently scale its compute or storage resources.

[0003] To address this need, the storage-computing separation architecture has been proposed as a new distributed system architecture. In this architecture, computing and storage logic are distributed across independent nodes, primarily compute nodes and storage nodes. Compute nodes are responsible for data computation and processing, while storage nodes are responsible for data storage and management. Compute nodes can be dynamically added or removed as needed to accommodate varying computing loads; similarly, storage nodes can dynamically expand storage capacity to accommodate varying data sizes. This system architecture improves the elasticity and scalability of distributed systems. Cloud-native databases such as Amazon Aurora, Apple FoundationDB, PolarDB, and Spanner have adopted it as their system architecture of choice.

[0004] In a database with separate storage and computation, distributed transactions must be executed and submitted between the compute nodes responsible for transaction execution and the storage nodes responsible for data storage, which requires extensive cross-node communication. Specifically, when a compute node executes a transaction, the read operations within the transaction need to read data from the remote storage node. When a compute node commits a transaction, the existing distributed commit protocol requires multiple network round trips to submit a single transaction. This architectural feature results in extended transaction execution time, a longer window for acquiring locks during transaction submission, an increased probability of conflicts between transactions, and reduced system throughput.

[0005] To reduce network traffic between compute nodes and storage nodes, there are two straightforward approaches. The first is to use local caches on compute nodes to reduce network overhead during the transaction execution phase. Compute nodes store frequently accessed storage node data in local caches, enabling them to quickly retrieve the data the next time it is accessed, without having to transfer data over the network again. The second approach is to use batch processing to reduce network overhead during the transaction commit phase. Compute nodes use batch processing to group multiple transactions into a single batch for commit, thereby spreading network overhead. However, both of these approaches can significantly increase transaction rollback overhead in distributed databases with storage and compute separation, primarily for two reasons. First, when transaction write operations executed on multiple compute nodes conflict, the compute nodes' local caches may contain expired or invalid data. When a transaction is ready to commit, it will be rolled back due to reading expired data, incurring additional overhead. Second, batch processing reduces the granularity of transaction conflicts from individual transactions to batched transactions, further increasing the likelihood of conflicts between transactions and exacerbating the overhead of transaction rollbacks.

[0006] In response to the above problems, the existing Amazon Aurora database uses local caching and batch commit, but it is limited to only one node in its transaction processing cluster that can be responsible for data updates, so that the local cache on the writer's computing node will not be invalidated, and there will be no conflicts between different computing nodes. However, such a design is severely limited by the performance of a single writer. The Apple FoundationDB database directly chooses not to set up a local cache on the computing node, and remote reading is still required during the transaction execution process. In response to the limitations of the above system, the present invention proposes a distributed database with storage and computing separation, which can simultaneously utilize local caching and batch commit, and innovatively uses a transaction repair mechanism to reduce the problem of excessive distributed transaction rollback overhead caused by caching and batch processing.

[0007] Patent document CN106874076B discloses a distributed transaction processing method, comprising: one of multiple transaction processing nodes receives a transaction initiation request and divides the transaction corresponding to the transaction initiation request into several sub-transactions; one of the transaction processing nodes executes a predetermined sub-transaction, writes the remaining sub-transactions to a local database, and publishes the sub-transactions in the local database to a message queue; the message queue pushes the sub-transactions to the remaining transaction processing nodes, writing the corresponding sub-transactions to the local databases of the remaining transaction processing nodes, and upon receiving a success indicator from the remaining transaction processing nodes, deletes the corresponding sub-transactions from the queue; the remaining transaction processing nodes retrieve and execute the corresponding sub-transactions from their respective local databases, and upon completion of execution, deletes the corresponding sub-transactions from the local databases. However, this invention does not utilize batch processing to reduce communication between computing nodes and storage nodes during transaction submission. Summary of the Invention

[0008] In view of the defects in the prior art, the purpose of the present invention is to provide a distributed concurrency control method and system based on fine-grained transaction repair.

[0009] According to the present invention, a distributed concurrency control method based on fine-grained transaction repair is provided, comprising:

[0010] Step S1: The computing node receives the user request and performs the transaction operation according to the request content;

[0011] Step S2: Based on the result of executing the transaction operation, the computing node commits the transaction locally and queues the locally committed transaction.

[0012] Step S3: When the local transaction queue of the computing node is full, all transactions in the queue are packaged into a batch transaction and the batch transaction is submitted globally to the storage node;

[0013] If a batch transaction fails verification during the global submission process, fine-grained transaction repair is performed to restore the batch transaction to a state where it can be submitted, and the batch transaction is then submitted again.

[0014] Preferably, in step S1:

[0015] The user request content includes the transaction logic to be executed and the corresponding transaction input; the transaction logic is pre-statically compiled into a data flow graph, and the nodes in the data flow graph represent a write operation or a read operation;

[0016] The executed transaction operations include read and write operations on data:

[0017] Read operation: The parameter is the key of the tuple to be read. During execution, a local shared lock is applied for the key to be read. If the lock acquisition fails, the transaction is rolled back. After the lock is successfully applied, the tuple is read from the local cache. If the local cache does not hit, a read request is sent to the storage node storing the tuple, and the read result is written back to the local cache. The read tuple is stored in the transaction's read set; the read set is the set of tuples read by the transaction read operation.

[0018] Write operation: The parameter is the key-value pair of the tuple to be written. During execution, a local mutex lock is applied for the written key. If the lock acquisition fails, the transaction is rolled back. After the lock is successfully applied, the tuple consisting of the written key-value pair and the identifier of the current transaction is stored in the transaction's write set. If the read and write operations cannot obtain the required lock due to lock conflicts, the current transaction needs to be rolled back and all the applied locks are released. The write set is a set that records the tuples written by the transaction write operation.

[0019] The tuple is the basic unit for storing data, and its content includes a key for indexing the tuple, representing the value and version of the user data; the tuple version is the identifier of the last transaction that modified the tuple; the transaction identifier is a 64-bit integer, and each transaction has a unique identifier.

[0020] Preferably, in step S2:

[0021] Step S2.1: For a completed transaction, write the tuples in the write set of the transaction to the local cache;

[0022] Step S2.2: Add the transaction to the local transaction queue of the computing node and release the local locks applied by the transaction for all tuples in its read set and write set;

[0023] The local transaction queue is a queue data structure maintained by each computing node, and the elements are unique identifiers of transactions.

[0024] Preferably, in step S3:

[0025] Step S3.1: After the number of transactions in the local transaction queue of the computing node reaches the threshold, all transactions are merged into a batch transaction. The read and write sets of all transactions are merged into the read set and write set of the batch transaction respectively, and duplicates are removed.

[0026] Step S3.2: The compute node sends a timestamp acquisition request to the timestamp server. The timestamp server combines the current timestamp units of the corresponding data segments accessed by the current batch transaction into a result timestamp based on the timestamp acquisition request. It also updates the timestamp units of the data segments based on the requested data segments and the corresponding access type, and returns the result timestamp to the compute node. After the compute node receives the returned timestamp, it proceeds to step S3.3.

[0027] Step S3.3: The compute node sends a prepare-to-commit request to the relevant storage node that stores the read set and write set tuples of the current batch transaction. After receiving the prepare-to-commit request, the storage node applies for the storage node's local shared lock and mutex lock for the read set and write set of the batch transaction based on the timestamp order between the corresponding batch transactions.

[0028] Step S3.4: The storage node verifies the read set of the batch transaction, persists the received prepare-to-commit request, and returns the verification result to the corresponding compute node. The compute node collects the verification results from the relevant storage nodes and returns them. If any storage node fails verification, the process proceeds to step S3.5. If all storage nodes succeed in verification, the process proceeds to step S3.6.

[0029] Step S3.5: The compute node begins fine-grained repair of the current batch transaction based on the latest tuple returned by the storage node that failed verification. After the transaction repair is complete, the compute node updates the local cache based on the repaired write set of the batch transaction and proceeds to step S3.6.

[0030] Step S3.6: The computing node sends a commit request to the storage node. After the storage node receives the commit request, if the corresponding batch is repaired, the repaired write set attached to the request is persisted. The storage node updates the local storage based on the write set of the batch transaction and releases all applied locks.

[0031] Preferably, the storage node is a logical storage node, which is composed of multiple physical storage nodes through state machine backup. Each physical storage node has the same initial state and processes requests from the computing nodes in the same order.

[0032] The prepare-to-commit request includes the read set and write set of the batch transaction, a timestamp, and transaction execution input;

[0033] Verification refers to comparing each tuple in the read set of the batch transaction being verified with the latest tuple with the same key locally on the storage node, comparing the versions of the two tuples. If the versions are the same, verification succeeds; otherwise, verification fails. In step S3.4, if verification fails, the storage node records the tuple that failed verification and returns the latest tuple with the same key locally on the storage node to the computing node.

[0034] The content of the submission confirmation request includes an identifier of the batch transaction to be submitted; each batch transaction is assigned a unique identifier;

[0035] The timestamp is a vector timestamp. Each element of the vector corresponds to a timestamp unit of a data segment. The timestamp unit is a pair of counters, seq and readers. The seq counter records the number of times the data segment has been modified by a batch transaction, and the readers counter records the number of batch transactions that have read the data segment since the current data segment was last modified.

[0036] The timestamp server is a server that provides timestamp services and stores timestamp units corresponding to all data segments. When processing timestamp acquisition requests from the same computing node, the timestamp server ensures that the corresponding batch transactions are processed in the order in which they are placed in the local transaction queue of the computing node. The timestamp server protects the atomicity of reading and updating timestamp units corresponding to data segments by acquiring locks. When updating the timestamp unit of a data segment in the server, if the target data segment is modified by the current batch transaction, the seq counter of the timestamp unit is incremented by one and the readers counter is cleared. If the target data segment is read by the current batch transaction, the readers counter of the timestamp unit is incremented by one.

[0037] The data segment is the basic unit for allocating timestamps. A data segment contains one or more tuples, and each node stores the mapping relationship between data segments and corresponding tuples. When a batch transaction modifies a data segment, it means that the tuple protected by the corresponding data segment exists in the write set of the batch transaction. When a batch transaction reads a data segment, it means that the tuple protected by the current data segment exists in the read set of the batch transaction, and the batch transaction does not modify the current data segment. When a batch transaction modifies or reads a data segment, it is said that the batch transaction has accessed the corresponding data segment. The timestamp acquisition request contains the set of data segments modified and the set of data segments read by the batch transaction.

[0038] The timestamp order is a partial order relation;

[0039] The fine-grained repair refers to the computing node re-executing the read or write operations that need to be re-executed. When repairing batch transactions, the computing node repairs each transaction in the current batch transaction in sequence according to the order in which the transactions are locally submitted. When repairing transactions, the computing node traverses the operation nodes in the data flow graph corresponding to the transaction logic according to the topological order. For a read operation, the computing node records the tuple read by the operation when the transaction is executed. During repair, the tuple recorded during execution is compared with the real-time tuple information of the current transaction repair process. If the tuple read by the current read operation is different, the current read operation is re-executed, and all operations that depend on the current read operation are also re-executed.

[0040] According to the present invention, a distributed concurrency control system based on fine-grained transaction repair is provided, comprising:

[0041] Module M1: The computing node receives user requests and executes transaction operations based on the request content;

[0042] Module M2: Based on the results of executing the transaction operation, the computing node commits the transaction locally and queues the locally submitted transaction.

[0043] Module M3: When the local transaction queue of the computing node is full, all transactions in the queue are packaged into a batch transaction and submitted to the storage node globally;

[0044] If a batch transaction fails verification during the global submission process, fine-grained transaction repair is performed to restore the batch transaction to a state where it can be submitted, and the batch transaction is then submitted again.

[0045] Preferably, in the module M1:

[0046] The user request content includes the transaction logic to be executed and the corresponding transaction input; the transaction logic is pre-statically compiled into a data flow graph, and the nodes in the data flow graph represent a write operation or a read operation;

[0047] The executed transaction operations include read and write operations on data:

[0048] Read operation: The parameter is the key of the tuple to be read. During execution, a local shared lock is applied for the key to be read. If the lock acquisition fails, the transaction is rolled back. After the lock is successfully applied, the tuple is read from the local cache. If the local cache does not hit, a read request is sent to the storage node storing the tuple, and the read result is written back to the local cache. The read tuple is stored in the transaction's read set; the read set is the set of tuples read by the transaction read operation.

[0049] Write operation: The parameter is the key-value pair of the tuple to be written. During execution, a local mutex lock is applied for the written key. If the lock acquisition fails, the transaction is rolled back. After the lock is successfully applied, the tuple consisting of the written key-value pair and the identifier of the current transaction is stored in the transaction's write set. If the read and write operations cannot obtain the required lock due to lock conflicts, the current transaction needs to be rolled back and all the applied locks are released. The write set is a set that records the tuples written by the transaction write operation.

[0050] The tuple is the basic unit for storing data, and its content includes a key for indexing the tuple, representing the value and version of the user data; the tuple version is the identifier of the last transaction that modified the tuple; the transaction identifier is a 64-bit integer, and each transaction has a unique identifier.

[0051] Preferably, in the module M2:

[0052] Module M2.1: For completed transactions, write the tuples in the write set of the transaction to the local cache;

[0053] Module M2.2: Add the transaction to the local transaction queue of the computing node and release the local locks applied by the transaction for all tuples in its read set and write set;

[0054] The local transaction queue is a queue data structure maintained by each computing node, and the elements are unique identifiers of transactions.

[0055] Preferably, in the module M3:

[0056] Module M3.1: When the number of transactions in the local transaction queue of a compute node reaches a threshold, all transactions are merged into a batch transaction. The read and write sets of all transactions are merged into the read and write sets of the batch transaction, and duplicates are removed.

[0057] Module M3.2: The compute node sends a timestamp request to the timestamp server. The timestamp server combines the current timestamp units of the corresponding data segments accessed by the current batch transaction into a result timestamp based on the timestamp request. It then updates the timestamp units of the data segments based on the requested data segments and the corresponding access type, and returns the result timestamp to the compute node. After receiving the returned timestamp, the compute node enters module M3.3.

[0058] Module M3.3: The compute node sends a prepare-commit request to the relevant storage node that stores the read and write set tuples of the current batch transaction. After receiving the prepare-commit request, the storage node applies for the storage node's local shared lock and mutex lock for the read and write sets of the batch transaction based on the timestamp order between the corresponding batch transactions.

[0059] Module M3.4: The storage node verifies the read set of the batch transaction, persists the received prepare-to-commit request, and returns the verification result to the corresponding compute node. The compute node collects the verification results from the relevant storage nodes and returns them. If any storage node verification fails, the process proceeds to module M3.5. If all storage nodes succeed in verification, the process proceeds to module M3.6.

[0060] Module M3.5: The compute node begins fine-grained repair of the current batch transaction based on the latest tuple returned by the storage node that failed verification. After the transaction repair is complete, the compute node updates the local cache based on the repaired write set of the batch transaction and enters module M3.6.

[0061] Module M3.6: The compute node sends a commit request to the storage node. After the storage node receives the commit request, if the corresponding batch is repaired, it persists the repaired write set included in the request. The storage node updates the local storage based on the write set of the batch transaction and releases all applied locks.

[0062] Preferably, the storage node is a logical storage node, which is composed of multiple physical storage nodes through state machine backup. Each physical storage node has the same initial state and processes requests from the computing nodes in the same order.

[0063] The prepare-to-commit request includes the read set and write set of the batch transaction, a timestamp, and transaction execution input;

[0064] Verification refers to comparing each tuple in the read set of the batch transaction being verified with the latest tuple with the same key locally on the storage node, comparing the versions of the two tuples. If the versions are the same, verification succeeds; otherwise, verification fails. In module M3.4, when verification fails, the storage node records the tuple that failed verification and returns the latest tuple with the same key locally on the storage node to the computing node.

[0065] The content of the submission confirmation request includes an identifier of the batch transaction to be submitted; each batch transaction is assigned a unique identifier;

[0066] The timestamp is a vector timestamp. Each element of the vector corresponds to a timestamp unit of a data segment. The timestamp unit is a pair of counters, seq and readers. The seq counter records the number of times the data segment has been modified by a batch transaction, and the readers counter records the number of batch transactions that have read the data segment since the current data segment was last modified.

[0067] The timestamp server is a server that provides timestamp services and stores timestamp units corresponding to all data segments. When processing timestamp acquisition requests from the same computing node, the timestamp server ensures that the corresponding batch transactions are processed in the order in which they are placed in the local transaction queue of the computing node. The timestamp server protects the atomicity of reading and updating timestamp units corresponding to data segments by acquiring locks. When updating the timestamp unit of a data segment in the server, if the target data segment is modified by the current batch transaction, the seq counter of the timestamp unit is incremented by one and the readers counter is cleared. If the target data segment is read by the current batch transaction, the readers counter of the timestamp unit is incremented by one.

[0068] The data segment is the basic unit for allocating timestamps. A data segment contains one or more tuples, and each node stores the mapping relationship between data segments and corresponding tuples. When a batch transaction modifies a data segment, it means that the tuple protected by the corresponding data segment exists in the write set of the batch transaction. When a batch transaction reads a data segment, it means that the tuple protected by the current data segment exists in the read set of the batch transaction, and the batch transaction does not modify the current data segment. When a batch transaction modifies or reads a data segment, it is said that the batch transaction has accessed the corresponding data segment. The timestamp acquisition request contains the set of data segments modified and the set of data segments read by the batch transaction.

[0069] The timestamp order is a partial order relation;

[0070] The fine-grained repair refers to the computing node re-executing the read or write operations that need to be re-executed. When repairing batch transactions, the computing node repairs each transaction in the current batch transaction in sequence according to the order in which the transactions are locally submitted. When repairing transactions, the computing node traverses the operation nodes in the data flow graph corresponding to the transaction logic according to the topological order. For a read operation, the computing node records the tuple read by the operation when the transaction is executed. During repair, the tuple recorded during execution is compared with the real-time tuple information of the current transaction repair process. If the tuple read by the current read operation is different, the current read operation is re-executed, and all operations that depend on the current read operation are also re-executed.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] 1. The present invention utilizes the local cache of computing nodes to reduce the communication caused by the operation of reading storage node data during transaction execution, thereby reducing transaction execution latency;

[0073] 2. The present invention uses batch processing to reduce the communication between computing nodes and storage nodes during transaction submission, thereby amortizing the network overhead of the global submission phase to multiple transactions in a batch transaction, thereby reducing the average network overhead of a single transaction;

[0074] 3. The present invention performs a two-phase hierarchical commit of transactions, local and remote. First, locks are used to coordinate local transactions on compute nodes, and then transactions from different compute nodes are coordinated using a verification-based method. This eliminates the need for conflict coordination between local transactions on remote storage nodes, thus avoiding the network message overhead of handling conflicts between local transactions on compute nodes.

[0075] 4. This invention allows locally committed transactions to read updates from transactions that have not been globally committed, improving throughput. Locally committed transactions make their write operations visible to other transactions running on the same node, so they do not have to wait for global commits, improving system parallelism.

[0076] 5. This invention uses fine-grained re-execution to repair outdated or invalid reads, rather than the traditional rollback and retry of the entire transaction. These techniques can reduce the cost of conflicts and make the system more scalable;

[0077] 6. The present invention uses a timestamp server to determine the global transaction submission order, avoiding the possibility of inconsistent processing order of different transactions on different storage nodes during global submission;

[0078] 7. The present invention adopts a partially ordered timestamp design. Compared with using fully ordered timestamps for all transactions, it enables the global submission of two batch transactions without conflict to be performed simultaneously, thereby improving the parallelism of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0080] Figure 1 The flowchart of the distributed concurrency control method based on fine-grained transaction repair is shown in Figure 2. DETAILED DESCRIPTION

[0081] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0082] Example 1:

[0083] The present invention provides a distributed concurrency control method and system based on fine-grained transaction repair, which is mainly aimed at distributed storage systems with a storage-computation separation architecture in which the computing nodes responsible for transaction processing are separated from the storage nodes responsible for data storage. By using a near-computing cache, the latency overhead of computing nodes reading remote data is reduced; by batch processing, distributed commits are performed for transactions on computing nodes, and the communication overhead caused by distributed commits is shared; by allowing transactions to read updates of transactions on the same computing node to complete the commit, the computing power of the computing nodes is fully utilized. Compared with existing work, the main difference of the present invention is that the use of a fine-grained transaction repair mechanism can greatly reduce the problems of increased distributed transaction conflicts and increased rollback overhead caused by cache and batch processing technologies; by adopting hierarchical distributed commits, there is no need for remote storage nodes to be responsible for handling transaction conflicts within computing nodes, which reduces the transaction conflict processing overhead; a dedicated timestamp server is used to determine the commit order of transactions, and a partially ordered timestamp is used to improve the parallelism of transaction commits.

[0084] According to the present invention, a distributed concurrency control method based on fine-grained transaction repair is provided, such as Figure 1 As shown, including:

[0085] Step S1: The computing node receives the user request and performs the transaction operation according to the request content;

[0086] Specifically, in step S1:

[0087] The user request content includes the transaction logic to be executed and the corresponding transaction input; the transaction logic is pre-statically compiled into a data flow graph, and the nodes in the data flow graph represent a write operation or a read operation;

[0088] The executed transaction operations include read and write operations on data:

[0089] Read operation: The parameter is the key of the tuple to be read. During execution, a local shared lock is applied for the key to be read. If the lock acquisition fails, the transaction is rolled back. After the lock is successfully applied, the tuple is read from the local cache. If the local cache does not hit, a read request is sent to the storage node storing the tuple, and the read result is written back to the local cache. The read tuple is stored in the transaction's read set; the read set is the set of tuples read by the transaction read operation.

[0090] Write operation: The parameter is the key-value pair of the tuple to be written. During execution, a local mutex lock is applied for the written key. If the lock acquisition fails, the transaction is rolled back. After the lock is successfully applied, the tuple consisting of the written key-value pair and the identifier of the current transaction is stored in the transaction's write set. If the read and write operations cannot obtain the required lock due to lock conflicts, the current transaction needs to be rolled back and all the applied locks are released. The write set is a set that records the tuples written by the transaction write operation.

[0091] The tuple is the basic unit for storing data, and its content includes a key for indexing the tuple, representing the value and version of the user data; the tuple version is the identifier of the last transaction that modified the tuple; the transaction identifier is a 64-bit integer, and each transaction has a unique identifier.

[0092] Step S2: Based on the result of executing the transaction operation, the computing node commits the transaction locally and queues the locally committed transaction.

[0093] Specifically, in step S2:

[0094] Step S2.1: For a completed transaction, write the tuples in the write set of the transaction to the local cache;

[0095] Step S2.2: Add the transaction to the local transaction queue of the computing node and release the local locks applied by the transaction for all tuples in its read set and write set;

[0096] The local transaction queue is a queue data structure maintained by each computing node, and the elements are unique identifiers of transactions.

[0097] Step S3: When the local transaction queue of the computing node is full, all transactions in the queue are packaged into a batch transaction and the batch transaction is submitted globally to the storage node;

[0098] If a batch transaction fails verification during the global submission process, fine-grained transaction repair is performed to restore the batch transaction to a state where it can be submitted, and the batch transaction is then submitted again.

[0099] Specifically, in step S3:

[0100] Step S3.1: After the number of transactions in the local transaction queue of the computing node reaches the threshold, all transactions are merged into a batch transaction. The read and write sets of all transactions are merged into the read set and write set of the batch transaction respectively, and duplicates are removed.

[0101] Step S3.2: The compute node sends a timestamp acquisition request to the timestamp server. The timestamp server combines the current timestamp units of the corresponding data segments accessed by the current batch transaction into a result timestamp based on the timestamp acquisition request. It also updates the timestamp units of the data segments based on the requested data segments and the corresponding access type, and returns the result timestamp to the compute node. After the compute node receives the returned timestamp, it proceeds to step S3.3.

[0102] Step S3.3: The compute node sends a prepare-to-commit request to the relevant storage node that stores the read set and write set tuples of the current batch transaction. After receiving the prepare-to-commit request, the storage node applies for the storage node's local shared lock and mutex lock for the read set and write set of the batch transaction based on the timestamp order between the corresponding batch transactions.

[0103] Step S3.4: The storage node verifies the read set of the batch transaction, persists the received prepare-to-commit request, and returns the verification result to the corresponding compute node. The compute node collects the verification results from the relevant storage nodes and returns them. If any storage node fails verification, the process proceeds to step S3.5. If all storage nodes succeed in verification, the process proceeds to step S3.6.

[0104] Step S3.5: The compute node begins fine-grained repair of the current batch transaction based on the latest tuple returned by the storage node that failed verification. After the transaction repair is complete, the compute node updates the local cache based on the repaired write set of the batch transaction and proceeds to step S3.6.

[0105] Step S3.6: The computing node sends a commit request to the storage node. After the storage node receives the commit request, if the corresponding batch is repaired, the repaired write set attached to the request is persisted. The storage node updates the local storage based on the write set of the batch transaction and releases all applied locks.

[0106] Specifically, the storage node is a logical storage node, which is composed of multiple physical storage nodes through state machine backup. Each physical storage node has the same initial state and processes requests from computing nodes in the same order;

[0107] The prepare-to-commit request includes the read set and write set of the batch transaction, a timestamp, and transaction execution input;

[0108] Verification refers to comparing each tuple in the read set of the batch transaction being verified with the latest tuple with the same key locally on the storage node, comparing the versions of the two tuples. If the versions are the same, verification succeeds; otherwise, verification fails. In step S3.4, if verification fails, the storage node records the tuple that failed verification and returns the latest tuple with the same key locally on the storage node to the computing node.

[0109] The content of the submission confirmation request includes an identifier of the batch transaction to be submitted; each batch transaction is assigned a unique identifier;

[0110] The timestamp is a vector timestamp. Each element of the vector corresponds to a timestamp unit of a data segment. The timestamp unit is a pair of counters, seq and readers. The seq counter records the number of times the data segment has been modified by a batch transaction, and the readers counter records the number of batch transactions that have read the data segment since the current data segment was last modified.

[0111] The timestamp server is a server that provides timestamp services and stores timestamp units corresponding to all data segments. When processing timestamp acquisition requests from the same computing node, the timestamp server ensures that the corresponding batch transactions are processed in the order in which they are placed in the local transaction queue of the computing node. The timestamp server protects the atomicity of reading and updating timestamp units corresponding to data segments by acquiring locks. When updating the timestamp unit of a data segment in the server, if the target data segment is modified by the current batch transaction, the seq counter of the timestamp unit is incremented by one and the readers counter is cleared. If the target data segment is read by the current batch transaction, the readers counter of the timestamp unit is incremented by one.

[0112] The data segment is the basic unit for allocating timestamps. A data segment contains one or more tuples, and each node stores the mapping relationship between data segments and corresponding tuples. When a batch transaction modifies a data segment, it means that the tuple protected by the corresponding data segment exists in the write set of the batch transaction. When a batch transaction reads a data segment, it means that the tuple protected by the current data segment exists in the read set of the batch transaction, and the batch transaction does not modify the current data segment. When a batch transaction modifies or reads a data segment, it is said that the batch transaction has accessed the corresponding data segment. The timestamp acquisition request contains the set of data segments modified and the set of data segments read by the batch transaction.

[0113] The timestamp order is a partial order relation;

[0114] The fine-grained repair refers to the computing node re-executing the read or write operations that need to be re-executed. When repairing batch transactions, the computing node repairs each transaction in the current batch transaction in sequence according to the order in which the transactions are locally submitted. When repairing transactions, the computing node traverses the operation nodes in the data flow graph corresponding to the transaction logic according to the topological order. For a read operation, the computing node records the tuple read by the operation when the transaction is executed. During repair, the tuple recorded during execution is compared with the real-time tuple information of the current transaction repair process. If the tuple read by the current read operation is different, the current read operation is re-executed, and all operations that depend on the current read operation are also re-executed.

[0115] Example 2:

[0116] Example 2 is a preferred example of Example 1 and is used to illustrate the present invention in more detail.

[0117] The present invention also provides a distributed concurrency control system based on fine-grained transaction repair. The distributed concurrency control system based on fine-grained transaction repair can be implemented by executing the process steps of the distributed concurrency control method based on fine-grained transaction repair, that is, those skilled in the art can understand the distributed concurrency control method based on fine-grained transaction repair as an optimal implementation method of the distributed concurrency control system based on fine-grained transaction repair.

[0118] According to the present invention, a distributed concurrency control system based on fine-grained transaction repair is provided, comprising:

[0119] Module M1: The computing node receives user requests and executes transaction operations based on the request content;

[0120] Specifically, in the module M1:

[0121] The user request content includes the transaction logic to be executed and the corresponding transaction input; the transaction logic is pre-statically compiled into a data flow graph, and the nodes in the data flow graph represent a write operation or a read operation;

[0122] The executed transaction operations include read and write operations on data:

[0123] Read operation: The parameter is the key of the tuple to be read. During execution, a local shared lock is applied for the key to be read. If the lock acquisition fails, the transaction is rolled back. After the lock is successfully applied, the tuple is read from the local cache. If the local cache does not hit, a read request is sent to the storage node storing the tuple, and the read result is written back to the local cache. The read tuple is stored in the transaction's read set; the read set is the set of tuples read by the transaction read operation.

[0124] Write operation: The parameter is the key-value pair of the tuple to be written. During execution, a local mutex lock is applied for the written key. If the lock acquisition fails, the transaction is rolled back. After the lock is successfully applied, the tuple consisting of the written key-value pair and the identifier of the current transaction is stored in the transaction's write set. If the read and write operations cannot obtain the required lock due to lock conflicts, the current transaction needs to be rolled back and all the applied locks are released. The write set is a set that records the tuples written by the transaction write operation.

[0125] The tuple is the basic unit for storing data, and its content includes a key for indexing the tuple, representing the value and version of the user data; the tuple version is the identifier of the last transaction that modified the tuple; the transaction identifier is a 64-bit integer, and each transaction has a unique identifier.

[0126] Module M2: Based on the results of executing the transaction operation, the computing node commits the transaction locally and queues the locally submitted transaction.

[0127] Specifically, in the module M2:

[0128] Module M2.1: For completed transactions, write the tuples in the write set of the transaction to the local cache;

[0129] Module M2.2: Add the transaction to the local transaction queue of the computing node and release the local locks applied by the transaction for all tuples in its read set and write set;

[0130] The local transaction queue is a queue data structure maintained by each computing node, and the elements are unique identifiers of transactions.

[0131] Module M3: When the local transaction queue of the computing node is full, all transactions in the queue are packaged into a batch transaction and submitted to the storage node globally;

[0132] If a batch transaction fails verification during the global submission process, fine-grained transaction repair is performed to restore the batch transaction to a state where it can be submitted, and the batch transaction is then submitted again.

[0133] Specifically, in the module M3:

[0134] Module M3.1: When the number of transactions in the local transaction queue of a compute node reaches a threshold, all transactions are merged into a batch transaction. The read and write sets of all transactions are merged into the read and write sets of the batch transaction, and duplicates are removed.

[0135] Module M3.2: The compute node sends a timestamp request to the timestamp server. The timestamp server combines the current timestamp units of the corresponding data segments accessed by the current batch transaction into a result timestamp based on the timestamp request. It then updates the timestamp units of the data segments based on the requested data segments and the corresponding access type, and returns the result timestamp to the compute node. After receiving the returned timestamp, the compute node enters module M3.3.

[0136] Module M3.3: The compute node sends a prepare-commit request to the relevant storage node that stores the read and write set tuples of the current batch transaction. After receiving the prepare-commit request, the storage node applies for the storage node's local shared lock and mutex lock for the read and write sets of the batch transaction based on the timestamp order between the corresponding batch transactions.

[0137] Module M3.4: The storage node verifies the read set of the batch transaction, persists the received prepare-to-commit request, and returns the verification result to the corresponding compute node. The compute node collects the verification results from the relevant storage nodes and returns them. If any storage node verification fails, the process proceeds to module M3.5. If all storage nodes succeed in verification, the process proceeds to module M3.6.

[0138] Module M3.5: The compute node begins fine-grained repair of the current batch transaction based on the latest tuple returned by the storage node that failed verification. After the transaction repair is complete, the compute node updates the local cache based on the repaired write set of the batch transaction and enters module M3.6.

[0139] Module M3.6: The compute node sends a commit request to the storage node. After the storage node receives the commit request, if the corresponding batch is repaired, it persists the repaired write set included in the request. The storage node updates the local storage based on the write set of the batch transaction and releases all applied locks.

[0140] Specifically, the storage node is a logical storage node, which is composed of multiple physical storage nodes through state machine backup. Each physical storage node has the same initial state and processes requests from computing nodes in the same order;

[0141] The prepare-to-commit request includes the read set and write set of the batch transaction, a timestamp, and transaction execution input;

[0142] Verification refers to comparing each tuple in the read set of the batch transaction being verified with the latest tuple with the same key locally on the storage node, comparing the versions of the two tuples. If the versions are the same, verification succeeds; otherwise, verification fails. In module M3.4, when verification fails, the storage node records the tuple that failed verification and returns the latest tuple with the same key locally on the storage node to the computing node.

[0143] The content of the submission confirmation request includes an identifier of the batch transaction to be submitted; each batch transaction is assigned a unique identifier;

[0144] The timestamp is a vector timestamp. Each element of the vector corresponds to a timestamp unit of a data segment. The timestamp unit is a pair of counters, seq and readers. The seq counter records the number of times the data segment has been modified by a batch transaction, and the readers counter records the number of batch transactions that have read the data segment since the current data segment was last modified.

[0145] The timestamp server is a server that provides timestamp services and stores timestamp units corresponding to all data segments. When processing timestamp acquisition requests from the same computing node, the timestamp server ensures that the corresponding batch transactions are processed in the order in which they are placed in the local transaction queue of the computing node. The timestamp server protects the atomicity of reading and updating timestamp units corresponding to data segments by acquiring locks. When updating the timestamp unit of a data segment in the server, if the target data segment is modified by the current batch transaction, the seq counter of the timestamp unit is incremented by one and the readers counter is cleared. If the target data segment is read by the current batch transaction, the readers counter of the timestamp unit is incremented by one.

[0146] The data segment is the basic unit for allocating timestamps. A data segment contains one or more tuples, and each node stores the mapping relationship between data segments and corresponding tuples. When a batch transaction modifies a data segment, it means that the tuple protected by the corresponding data segment exists in the write set of the batch transaction. When a batch transaction reads a data segment, it means that the tuple protected by the current data segment exists in the read set of the batch transaction, and the batch transaction does not modify the current data segment. When a batch transaction modifies or reads a data segment, it is said that the batch transaction has accessed the corresponding data segment. The timestamp acquisition request contains the set of data segments modified and the set of data segments read by the batch transaction.

[0147] The timestamp order is a partial order relation;

[0148] The fine-grained repair refers to the computing node re-executing the read or write operations that need to be re-executed. When repairing batch transactions, the computing node repairs each transaction in the current batch transaction in sequence according to the order in which the transactions are locally submitted. When repairing transactions, the computing node traverses the operation nodes in the data flow graph corresponding to the transaction logic according to the topological order. For a read operation, the computing node records the tuple read by the operation when the transaction is executed. During repair, the tuple recorded during execution is compared with the real-time tuple information of the current transaction repair process. If the tuple read by the current read operation is different, the current read operation is re-executed, and all operations that depend on the current read operation are also re-executed.

[0149] Example 3:

[0150] Example 3 is a preferred example of Example 1 and is used to illustrate the present invention in more detail.

[0151] The distributed concurrency control method based on fine-grained transaction repair provided by the present invention includes:

[0152] Step S1: The computing node receives the user request and performs the transaction operation according to the request content.

[0153] Step S2: Based on the transaction execution result, the computing node submits the transaction locally and queues the locally submitted transaction.

[0154] Step S3: When the local transaction queue of the compute node is full, all transactions in the queue are packaged into a batch transaction and submitted globally to the storage node. If the transaction fails verification during the global submission process, the transaction is restored to a committable state through fine-grained transaction repair, and then the transaction is submitted again.

[0155] The storage node is actually a logical storage node, which is composed of multiple physical storage nodes through state machine backup. Each physical storage node has the same initial state and deterministically processes requests from computing nodes in the same order, thereby achieving the same data state and playing the role of backup fault tolerance.

[0156] The user request content includes the transaction logic to be executed and the corresponding transaction input. The transaction logic is pre-statically compiled into a data flow graph, where a node represents a write or read operation. If the input of operation 1 is the output of operation 2, there will be a directed edge from operation 2 to operation 1 in the data flow graph. If there is a path in the data flow graph from operation 1 to operation 2, operation 2 is said to be dependent on operation 1.

[0157] Preferably, the transaction operations executed in step S1 may include read operations and write operations on data. Read operation: The parameter is the key of the tuple to be read. During execution, a local shared lock is first applied for the key to be read. If the lock fails to be obtained, the transaction is rolled back. If the lock is successfully applied, an attempt is made to read the tuple from the local cache. If the local cache does not hit, a read request is sent to the storage node storing the tuple, and the read result is written back to the local cache. Finally, the read tuple is stored in the read set of the transaction. Write operation: The parameter is the key-value pair of the tuple to be written. During execution, a local mutex lock is first applied for the key to be written. If the lock fails to be obtained, the transaction is rolled back. After the lock is successfully applied, the tuple consisting of the written key-value pair and the identifier of the current transaction is stored in the write set of the transaction. If the read operation and the write operation cannot apply for the required lock due to lock conflict, the current transaction needs to be rolled back and all applied locks are released.

[0158] The tuple is the basic unit for storing data in this invention. Its content includes a key for indexing the tuple, a value representing user data, and a version. The version of a tuple is the identifier of the last transaction that modified the tuple. The transaction identifier is a 64-bit integer, and each transaction in the system has a unique identifier. The read set is a set that records the tuples read by the transaction read operation. The write set is a set that records the tuples written by the transaction write operation.

[0159] Preferably, the step S2 adopts:

[0160] Step S2.1: For a completed transaction, write the tuples in the write set of the transaction into the local cache.

[0161] Step S2.2: Add the transaction to the local transaction queue of the computing node, and then release the local locks applied by the transaction for all tuples in its read set and write set.

[0162] The local transaction queue is a queue data structure maintained by each computing node. The elements are unique identifiers of transactions, which are used to determine the local submission order of local transactions.

[0163] Preferably, the step S3 adopts:

[0164] Step S3.1: After the number of transactions in the local transaction queue of the computing node reaches the threshold, all transactions are merged into a batch transaction, and the read and write sets of all transactions are merged into the read set and write set of the batch transaction respectively, and duplicates are removed.

[0165] Step S3.2: The compute node sends a timestamp request to the timestamp server. Based on the data segments accessed by the current batch transaction in the timestamp request, the timestamp server combines the current timestamp units of the corresponding data segments into a result timestamp. It then updates the timestamp units of the data segments based on the requested data segments and the corresponding access type, and returns the result timestamp to the compute node. After receiving the returned timestamp, the compute node proceeds to step S3.3.

[0166] Step S3.3: The compute node sends a prepare-commit request to the relevant storage node that stores the read and write set tuples for the current batch transaction. After receiving the prepare-commit request, the storage node applies for the storage node's local shared lock and mutex lock for the read and write sets of the batch transaction based on the timestamp order of the corresponding batch transactions.

[0167] Step S3.4: The storage node verifies the batch transaction's read set, persists the received prepare-to-commit request, and returns the verification result to the corresponding compute node. The compute node collects the verification results from the relevant storage nodes and returns them. If any storage node fails verification, the process proceeds to step S3.5. Otherwise, the process proceeds to step S3.6.

[0168] Step S3.5: The compute node begins fine-grained repair of the current batch transaction based on the latest tuple returned by the storage node that failed verification. After the transaction repair is complete, the compute node updates its local cache based on the repaired write set of the batch transaction and proceeds to step S3.6.

[0169] Step S3.6: The compute node sends a commit request to the storage node. After receiving the commit request, if the corresponding batch has been repaired, the storage node persists the repaired write set included in the request. The storage node then updates local storage based on the write set of the batch transaction and releases all requested locks.

[0170] The prepare-to-commit request includes a read set and a write set of a batch transaction, a timestamp, and a transaction execution input.

[0171] Verification specifically involves comparing each tuple in the read set of the batch transaction being verified with the latest tuple with the same key locally on the storage node. Specifically, the versions of the two tuples are compared. If the versions are the same, verification succeeds; otherwise, verification fails. In step S3.4, if verification fails, the storage node records the tuple that failed verification and returns the latest tuple with the same key locally on the storage node to the compute node.

[0172] The content of the submission confirmation request includes an identifier of the batch transaction to be submitted. Each batch transaction is assigned a unique identifier.

[0173] Preferably, the timestamp (TS) is a vector timestamp, and each element of the vector corresponds to a timestamp unit (TSU) of a data segment. The timestamp unit is a pair of counters<seq,readers> , the seq counter records the number of times the data segment has been modified by the batch transaction, and the readers counter records the number of batch transactions that have read the data segment since the current data segment was last modified. The timestamp server is a server that provides timestamp services and stores the timestamp units corresponding to all data segments. When processing timestamp acquisition requests from the same computing node, the timestamp server ensures that the corresponding batch transactions are processed in the order in which they appear in the local transaction queue of the computing node. The timestamp server protects the atomicity of reading and updating the timestamp units corresponding to the data segments by taking locks. When updating the timestamp unit of the data segment in the time server, if the target data segment is modified by the current batch transaction, the seq counter of the timestamp unit is incremented by one and the readers counter is cleared to zero; if the target data segment is read by the current batch transaction, the readers counter of the timestamp unit is incremented by one.

[0174] The data segment is the basic unit for allocating timestamp units. A data segment contains several tuples, and each node will save the mapping relationship between the data segment and the corresponding tuple. A batch transaction modifies a data segment, indicating that there are tuples protected by the corresponding data segment in the write set of the batch transaction. A batch transaction reads a data segment, indicating that there are tuples protected by the current data segment in the read set of the batch transaction, and the batch transaction does not modify the current data segment. When a batch transaction modifies or reads a data segment, it is said that the batch transaction accesses the corresponding data segment. The timestamp acquisition request contains the set of data segments modified by the batch transaction and the set of data segments read by the batch transaction.

[0175] The timestamp order is a partial order relationship. For two timestamps TS1 and TS2, TS1 precedes TS2 if and only if TS1 and TS2 access any same data segment, and the timestamp unit of TS1 on the common data segment precedes the timestamp unit of TS2 on the common data segment. In two cases, the timestamp unit TSU1<seq1,readers1> precedes the timestamp unit TSU2<seq2,readers2>. The first case is that the data segment corresponding to TSU2 is read by the current batch transaction, that is, when seq1 < seq2 || (seq1 == seq2 && readers1 == 0), TSU1 precedes TSU2; the second case is that the data segment corresponding to TSU2 is updated by the current batch transaction, that is, when seq1 < seq2 || (seq1 == seq2 && readers1 < readers2), TSU1 precedes TSU2.

[0176] Preferably, the fine-grained repair refers to the computing node selectively re-executing the read operations or write operations that need to be re-executed. When repairing a batch transaction, the computing node will repair each transaction in the current batch transaction in turn according to the order of local submission of the transaction. When repairing a transaction, the computing node will traverse the operation nodes in the data flow graph corresponding to the transaction logic according to the topological order. For a read operation, the computing node will record the tuple read by the operation during transaction execution, and compare the tuple recorded during execution with the real-time tuple information in the current transaction repair process during repair. If the tuple read by the current read operation is different, the current read operation needs to be re-executed, and all operations dependent on the current read operation also need to be re-executed.

[0177] Embodiment 4:

[0178] Embodiment 4 is a preferred example of Embodiment 1, and is used to illustrate the present invention more specifically.

[0179] ]>According to a distributed concurrency control method based on fine-grained transaction repair provided by the present invention, as Figure 1 shown, it includes:

[0180] Local execution step: The computing node receives the user request and executes the transaction operation according to the request content. Enter the local commit step.

[0181] Local update step: For completed transactions, write the tuples in the transaction's write set to the local cache. Then proceed to the local lock release step.

[0182] Local lock release step: The transaction is added to the local transaction queue of the compute node, and then the local locks applied by the transaction for all tuples in its read and write sets are released. The global commit step is then initiated.

[0183] Transaction merging step: When the number of transactions in the compute node's local transaction queue reaches the threshold, all transactions are merged into a batch transaction. The read and write sets of all transactions are merged into the batch transaction's read and write sets, respectively, and duplicates are removed. The timestamp acquisition step then begins.

[0184] Timestamp acquisition step: The compute node sends a timestamp acquisition request to the timestamp server. Based on the data segments accessed by the current batch transaction in the timestamp acquisition request, the timestamp server combines the current timestamp units of the corresponding data segments into a result timestamp. It then updates the timestamp units of the data segments based on the requested data segments and the corresponding access type, and finally returns the result timestamp to the compute node. After receiving the returned timestamp, the compute node proceeds to the lock request step.

[0185] Lock request step: The compute node sends a prepare-commit request to the relevant storage node that stores the read and write set tuples of the current batch transaction. After receiving the prepare-commit request, the storage node applies for the storage node-local shared lock and mutex lock for the read and write sets of the batch transaction based on the timestamp order of the corresponding batch transactions. Then, the verification step begins.

[0186] Verification step: Storage nodes verify the read set of the batch transaction, persist the received prepare-to-commit request, and then return the verification results to the corresponding compute nodes. The compute nodes collect the verification results from the relevant storage nodes and return them. If any storage node fails verification, the transaction proceeds to the transaction repair step. Otherwise, the global commit step proceeds.

[0187] Transaction repair step: The compute node begins fine-grained repair of the current batch transaction based on the latest tuples returned by the storage node that failed verification. After the transaction repair is complete, the compute node updates the local cache based on the repaired write set of the batch transaction and enters the global commit step.

[0188] Global commit step: The compute node sends a commit request to the storage node. After receiving the commit request, if the corresponding batch is repaired, the storage node persists the repaired write set included in the request. The storage node then updates local storage based on the batch transaction's write set and releases all requested locks.

[0189] The storage node is actually a logical storage node, which is composed of multiple physical storage nodes through state machine backup. Each physical storage node has the same initial state and deterministically processes requests from computing nodes in the same order, thereby achieving the same data state and playing the role of backup fault tolerance.

[0190] The user request content includes the transaction logic to be executed and the corresponding transaction input. The transaction logic is pre-statically compiled into a data flow graph, where a node represents a write or read operation. If the input of operation 1 is the output of operation 2, there will be a directed edge from operation 2 to operation 1 in the data flow graph. If there is a path in the data flow graph from operation 1 to operation 2, operation 2 is said to be dependent on operation 1.

[0191] Specifically, the transaction operations executed in the local execution step may include read operations and write operations on data. Read operation: The parameter is the key of the tuple to be read. During execution, a local shared lock is first applied for the key to be read. If the lock fails to be obtained, the transaction is rolled back. If the lock is successfully applied, an attempt is made to read the tuple from the local cache. If the local cache does not hit, a read request is sent to the storage node storing the tuple, and the read result is written back to the local cache. Finally, the read tuple is stored in the read set of the transaction. Write operation: The parameter is the key-value pair of the tuple to be written. During execution, a local mutex lock is first applied for the key to be written. If the lock fails to be obtained, the transaction is rolled back. After the lock is successfully applied, the tuple consisting of the written key-value pair and the identifier of the current transaction is stored in the write set of the transaction. If the read operation and the write operation cannot apply for the required lock due to lock conflict, the current transaction needs to be rolled back and all applied locks are released.

[0192] The tuple is the basic unit for storing data in this invention. Its content includes a key for indexing the tuple, a value representing user data, and a version. The version of a tuple is the identifier of the last transaction that modified the tuple. The transaction identifier is a 64-bit integer, and each transaction in the system has a unique identifier. The read set is a set that records the tuples read by the transaction read operation. The write set is a set that records the tuples written by the transaction write operation.

[0193] The local transaction queue is a queue data structure maintained by each computing node. The elements are unique identifiers of transactions, which are used to determine the local submission order of local transactions.

[0194] The prepare-to-commit request includes a read set and a write set of a batch transaction, a timestamp, and a transaction execution input.

[0195] Verification specifically involves comparing each tuple in the read set of the batch transaction being verified with the latest tuple with the same key stored locally on the storage node. Specifically, the versions of the two tuples are compared. If the versions are the same, verification succeeds; otherwise, verification fails. If verification fails, the storage node records the failed tuple and returns the latest tuple with the same key stored locally on the storage node to the compute node.

[0196] The content of the submission confirmation request includes an identifier of the batch transaction to be submitted. Each batch transaction is assigned a unique identifier.

[0197] Specifically, the timestamp (TS) is a vector timestamp, and each element of the vector corresponds to a timestamp unit (TSU) of a data segment. The timestamp unit is a pair of counters.<seq,readers> , the seq counter records the number of times the data segment has been modified by the batch transaction, and the readers counter records the number of batch transactions that have read the data segment since the current data segment was last modified. The timestamp server is a server that provides timestamp services and stores the timestamp units corresponding to all data segments. When processing timestamp acquisition requests from the same computing node, the timestamp server ensures that the corresponding batch transactions are processed in the order in which they appear in the local transaction queue of the computing node. The timestamp server protects the atomicity of reading and updating the timestamp units corresponding to the data segments by taking locks. When updating the timestamp unit of the data segment in the time server, if the target data segment is modified by the current batch transaction, the seq counter of the timestamp unit is incremented by one and the readers counter is cleared to zero; if the target data segment is read by the current batch transaction, the readers counter of the timestamp unit is incremented by one.

[0198] The data segment is the basic unit for allocating timestamp units. A data segment contains several tuples, and each node will save the mapping relationship between data segments and corresponding tuples. When a batch transaction modifies a data segment, it means that the tuple protected by the corresponding data segment exists in the write set of the batch transaction. When a batch transaction reads a data segment, it means that the tuple protected by the current data segment exists in the read set of the batch transaction, and the batch transaction does not modify the current data segment. When a batch transaction modifies or reads a data segment, it is said that the batch transaction accesses the corresponding data segment. The timestamp acquisition request contains the set of data segments modified by the batch transaction and the set of data segments read.

[0199] The timestamp order is a partial order relation. For two timestamps TS1 and TS2, TS1 precedes TS2 if and only if TS1 and TS2 access any same data segment, and the timestamp unit of TS1 on the common data segment precedes the timestamp unit of TS2 on the common data segment. In two cases, the timestamp unit TSU1<seq1,readers1> precedes the timestamp unit TSU2<seq2,readers2>. The first case is that the data segment corresponding to TSU2 is read by the current batch transaction, that is, when seq1 < seq2 || (seq1 == seq2 / \ readers1 == 0), TSU1 precedes TSU2; the second case is that the data segment corresponding to TSU2 is updated by the current batch transaction, that is, when seq1 < seq2 || (seq1 == seq2 / \ readers1 < readers2), TSU1 precedes TSU2.

[0200] Specifically, the fine-grained repair refers to that the computing node selectively re-executes the read operations or write operations that need to be re-executed. When repairing a batch transaction, the computing node will repair each transaction in the current batch transaction in turn according to the order of local submission of the transaction. When repairing a transaction, the computing node will traverse the operation nodes in the data flow graph corresponding to the transaction logic according to the topological order. For a read operation, the computing node will record the tuple read by the operation during transaction execution, and compare the tuple recorded during execution with the real-time tuple information of the current transaction repair process during repair. If the tuple read by the current read operation is different, the current read operation needs to be re-executed, and all operations depending on the current read operation also need to be re-executed. In the above fine-grained repair process, if the transaction reads or writes a tuple that does not exist in the original read set or write set, the current repaired transaction must be rolled back and its write set is cleared.

[0201] Those skilled in the art know that in addition to implementing the system, device and its various modules provided by the present invention in the form of pure computer-readable program code, the method steps can be logically programmed to make the system, device and its various modules provided by the present invention be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers, etc. to implement the same program. Therefore, the system, device and its various modules provided by the present invention can be regarded as a hardware component, and the modules included therein for implementing various programs can also be regarded as the structure within the hardware component; the modules for implementing various functions can also be regarded as either a software program for implementing the method or the structure within the hardware component.

[0202] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A distributed concurrency control method based on fine-grained transaction repair, characterized in that: include: Step S1: The computing node receives the user request and performs the transaction operation according to the request content; Step S2: Based on the result of executing the transaction operation, the computing node locally commits the transaction and queues the locally committed transaction; Step S3: When the local transaction queue of the computing node is full, all transactions in the queue are packaged into a batch transaction and the batch transaction is submitted globally to the storage node; If a batch transaction fails verification during the global submission process, fine-grained transaction repair is used to restore the batch transaction to a state where it can be submitted, and the batch transaction is then submitted again. Executing transaction operations includes reading and writing data; In step S2: Step S2.1: For a completed transaction, write the tuples in the write set of the transaction to the local cache; Step S2.2: Add the transaction to the local transaction queue of the computing node and release the local shared locks and mutex locks applied by the transaction for all tuples in its read set and write set; The local transaction queue is a queue data structure maintained by each computing node, and the elements are unique identifiers of transactions; In step S3: Step S3.1: After the number of transactions in the local transaction queue of the computing node reaches the threshold, all transactions are merged into a batch transaction. The read and write sets of all transactions are merged into the read set and write set of the batch transaction respectively, and duplicates are removed. Step S3.2: The compute node sends a timestamp acquisition request to the timestamp server. The timestamp server combines the current timestamp units of the corresponding data segments accessed by the current batch transaction into a result timestamp based on the timestamp acquisition request. It also updates the timestamp units of the data segments based on the requested data segments and the corresponding access type, and returns the result timestamp to the compute node. After the compute node receives the returned timestamp, it proceeds to step S3.

3. Step S3.3: The compute node sends a prepare-to-commit request to the relevant storage node that stores the read set and write set tuples of the current batch transaction. After receiving the prepare-to-commit request, the storage node applies for the storage node's local shared lock and mutex lock for the read set and write set of the batch transaction based on the timestamp order between the corresponding batch transactions. Step S3.4: The storage node verifies the read set of the batch transaction, persists the received prepare-to-commit request, and returns the verification result to the corresponding computing node; The computing node collects the verification results from the relevant storage nodes and returns them; If any storage node fails verification, proceed to step S3.5; if all storage nodes succeed in verification, proceed to step S3.6; Step S3.5: The compute node begins fine-grained repair of the current batch transaction based on the latest tuple returned by the storage node that failed verification. After the transaction repair is complete, the compute node updates the local cache based on the repaired write set of the batch transaction and proceeds to step S3.

6. Step S3.6: The compute node sends a commit request to the storage node. After receiving the commit request, if the corresponding batch is repaired, the storage node persists the repaired write set included in the request. The storage node updates the local storage based on the write set of the batch transaction and releases all local shared locks and local exclusive locks of the storage node that have been applied for. The storage node is a logical storage node, which is composed of multiple physical storage nodes through state machine backup. Each physical storage node has the same initial state and processes requests from computing nodes in the same order. The prepare-to-commit request includes the read set and write set of the batch transaction, a timestamp, and transaction execution input; Verification refers to comparing each tuple in the read set of the batch transaction being verified with the latest tuple with the same key locally on the storage node, comparing the versions of the two tuples. If the versions are the same, verification succeeds; otherwise, verification fails. In step S3.4, if verification fails, the storage node records the tuple that failed verification and returns the latest tuple with the same key locally on the storage node to the computing node. The content of the submission confirmation request includes the identifier of the batch transaction to be submitted; each batch transaction is assigned a unique identifier; The timestamp is a vector timestamp. Each element of the vector corresponds to a timestamp unit of a data segment. The timestamp unit is a pair of counters, seq and readers. The seq counter records the number of times the data segment has been modified by a batch transaction, and the readers counter records the number of batch transactions that have read the data segment since the current data segment was last modified. The timestamp server is a server that provides timestamp services and stores the timestamp units corresponding to all data segments. When processing timestamp acquisition requests from the same computing node, the timestamp server ensures that the corresponding batch transactions are processed in the order in which they are in the local transaction queue of the computing node. The timestamp server protects the atomicity of reading and updating the timestamp unit corresponding to the data segment by taking the local mutex of the timestamp server; When updating the timestamp unit of a data segment in the time server, if the target data segment is modified by the current batch transaction, the seq counter of the timestamp unit is incremented by one and the readers counter is cleared. If the target data segment is read by the current batch transaction, the readers counter of the timestamp unit is incremented by one; The data segment is the basic unit for allocating timestamps. A data segment contains one or more tuples, and each node stores the mapping relationship between data segments and corresponding tuples. When a batch transaction modifies a data segment, it means that the tuple protected by the corresponding data segment exists in the write set of the batch transaction. When a batch transaction reads a data segment, it means that the tuple protected by the current data segment exists in the read set of the batch transaction, and the batch transaction does not modify the current data segment. When a batch transaction modifies or reads a data segment, it is said that the batch transaction has accessed the corresponding data segment. The timestamp acquisition request contains the set of data segments modified and the set of data segments read by the batch transaction. The timestamp order is a partial order relation; The fine-grained repair refers to the computing node re-executing the read or write operations that need to be re-executed. When repairing batch transactions, the computing node repairs each transaction in the current batch transaction in sequence according to the order in which the transactions are locally submitted. When repairing transactions, the computing node traverses the operation nodes in the data flow graph corresponding to the transaction logic according to the topological order. For a read operation, the computing node records the tuple read by the operation when the transaction is executed. During repair, the tuple recorded during execution is compared with the real-time tuple information of the current transaction repair process. If the tuple read by the current read operation is different, the current read operation is re-executed, and all operations that depend on the current read operation are also re-executed.

2. The distributed concurrency control method based on fine-grained transaction repair according to claim 1 is characterized in that: In step S1: The user request content includes the transaction logic to be executed and the corresponding transaction input; the transaction logic is pre-statically compiled into a data flow graph, and the nodes in the data flow graph represent a write operation or a read operation; Transaction operations include reading and writing data: Read operation: The parameter is the key of the tuple to be read. During execution, a local shared lock is applied for the key to be read. If the local shared lock fails, the transaction is rolled back. After the local shared lock is successfully applied, the tuple is read from the local cache. If the local cache misses, a read request is sent to the storage node storing the tuple, and the read result is written back to the local cache, and the read tuple is stored in the transaction's read set; the read set is a set that records the tuples read by the transaction read operation; Write operation: The parameter is the key-value pair of the tuple to be written. During execution, a local mutex is applied for the written key. If the local mutex fails to be obtained, the transaction is rolled back. After the local mutex is successfully applied, the tuple consisting of the written key-value pair and the identifier of the current transaction is stored in the write set of the transaction. If the read operation cannot obtain the required local shared lock due to a local shared lock conflict, or if the write operation cannot obtain the required local mutex due to a local mutex conflict, the current transaction needs to be rolled back and all local mutexes and local shared locks applied by the current transaction are released. The write set is a set that records the write tuples of the transaction write operation; The tuple is the basic unit for storing data, and its content includes a key for indexing the tuple, representing the value and version of the user data; the tuple version is the identifier of the last transaction that modified the tuple; the transaction identifier is a 64-bit integer, and each transaction has a unique identifier.

3. A distributed concurrent control system based on fine-grained transaction repair, characterized in that: include: Module M1: The computing node receives user requests and executes transaction operations based on the request content; Module M2: Based on the results of the transaction operation, the computing node locally commits the transaction and queues the locally committed transaction. Module M3: When the local transaction queue of the computing node is full, all transactions in the queue are packaged into a batch transaction and submitted to the storage node globally; If a batch transaction fails verification during the global submission process, fine-grained transaction repair is used to restore the batch transaction to a state where it can be submitted, and the batch transaction is then submitted again. Executing transaction operations includes reading and writing data; In the module M2: Module M2.1: For completed transactions, write the tuples in the write set of the transaction to the local cache; Module M2.2: Adds the transaction to the local transaction queue of the compute node and releases the local shared locks and mutex locks applied by the transaction for all tuples in its read and write sets. The local transaction queue is a queue data structure maintained by each computing node, and the elements are unique identifiers of transactions; In the module M3: Module M3.1: When the number of transactions in the local transaction queue of a compute node reaches a threshold, all transactions are merged into a batch transaction. The read and write sets of all transactions are merged into the read and write sets of the batch transaction, and duplicates are removed. Module M3.2: The compute node sends a timestamp request to the timestamp server. The timestamp server combines the current timestamp units of the corresponding data segments accessed by the current batch transaction into a result timestamp based on the timestamp request. It then updates the timestamp units of the data segments based on the requested data segments and the corresponding access type, and returns the result timestamp to the compute node. After receiving the returned timestamp, the compute node enters module M3.

3. Module M3.3: The compute node sends a prepare-commit request to the relevant storage node that stores the read and write set tuples of the current batch transaction. After receiving the prepare-commit request, the storage node applies for the storage node's local shared lock and mutex lock for the read and write sets of the batch transaction based on the timestamp order between the corresponding batch transactions. Module M3.4: The storage node verifies the read set of the batch transaction, persists the received prepare-to-commit request, and returns the verification result to the corresponding compute node; The computing node collects the verification results from the relevant storage nodes and returns them; If any storage node fails verification, proceed to module M3.5; if all storage nodes succeed in verification, proceed to module M3.6; Module M3.5: The compute node begins fine-grained repair of the current batch transaction based on the latest tuple returned by the storage node that failed verification. After the transaction repair is complete, the compute node updates the local cache based on the repaired write set of the batch transaction and enters module M3.

6. Module M3.6: The compute node sends a commit request to the storage node. After receiving the commit request, if the corresponding batch is repaired, the storage node persists the repaired write set included in the request. The storage node updates the local storage based on the write set of the batch transaction and releases all local shared locks and local exclusive locks of the storage node that have been applied for. The storage node is a logical storage node, which is composed of multiple physical storage nodes through state machine backup. Each physical storage node has the same initial state and processes requests from computing nodes in the same order. The prepare-to-commit request includes the read set and write set of the batch transaction, a timestamp, and transaction execution input; Verification refers to comparing each tuple in the read set of the batch transaction being verified with the latest tuple with the same key locally on the storage node, comparing the versions of the two tuples. If the versions are the same, verification succeeds; otherwise, verification fails. In module M3.4, when verification fails, the storage node records the tuple that failed verification and returns the latest tuple with the same key locally on the storage node to the computing node. The content of the submission confirmation request includes the identifier of the batch transaction to be submitted; each batch transaction is assigned a unique identifier; The timestamp is a vector timestamp. Each element of the vector corresponds to a timestamp unit of a data segment. The timestamp unit is a pair of counters, seq and readers. The seq counter records the number of times the data segment has been modified by a batch transaction, and the readers counter records the number of batch transactions that have read the data segment since the current data segment was last modified. The timestamp server is a server that provides timestamp services and stores the timestamp units corresponding to all data segments. When processing timestamp acquisition requests from the same computing node, the timestamp server ensures that the corresponding batch transactions are processed in the order in which they are in the local transaction queue of the computing node. The timestamp server protects the atomicity of reading and updating the timestamp unit corresponding to the data segment by taking the local mutex of the timestamp server; When updating the timestamp unit of a data segment in the time server, if the target data segment is modified by the current batch transaction, the seq counter of the timestamp unit is incremented by one and the readers counter is cleared. If the target data segment is read by the current batch transaction, the readers counter of the timestamp unit is incremented by one; The data segment is the basic unit for allocating timestamps. A data segment contains one or more tuples, and each node stores the mapping relationship between data segments and corresponding tuples. When a batch transaction modifies a data segment, it means that the tuple protected by the corresponding data segment exists in the write set of the batch transaction. When a batch transaction reads a data segment, it means that the tuple protected by the current data segment exists in the read set of the batch transaction, and the batch transaction does not modify the current data segment. When a batch transaction modifies or reads a data segment, it is said that the batch transaction has accessed the corresponding data segment. The timestamp acquisition request contains the set of data segments modified and the set of data segments read by the batch transaction. The timestamp order is a partial order relation; The fine-grained repair refers to the computing node re-executing the read or write operations that need to be re-executed. When repairing batch transactions, the computing node repairs each transaction in the current batch transaction in sequence according to the order in which the transactions are locally submitted. When repairing transactions, the computing node traverses the operation nodes in the data flow graph corresponding to the transaction logic according to the topological order. For a read operation, the computing node records the tuple read by the operation when the transaction is executed. During repair, the tuple recorded during execution is compared with the real-time tuple information of the current transaction repair process. If the tuple read by the current read operation is different, the current read operation is re-executed, and all operations that depend on the current read operation are also re-executed.

4. The distributed concurrent control system based on fine-grained transaction repair according to claim 3, characterized in that: In the module M1: The user request content includes the transaction logic to be executed and the corresponding transaction input; the transaction logic is pre-statically compiled into a data flow graph, and the nodes in the data flow graph represent a write operation or a read operation; Transaction operations include reading and writing data: Read operation: The parameter is the key of the tuple to be read. During execution, a local shared lock is applied for the key to be read. If the local shared lock fails, the transaction is rolled back. After the local shared lock is successfully applied, the tuple is read from the local cache. If the local cache misses, a read request is sent to the storage node storing the tuple, and the read result is written back to the local cache, and the read tuple is stored in the transaction's read set; the read set is a set that records the tuples read by the transaction read operation; Write operation: The parameter is the key-value pair of the tuple to be written. During execution, a local mutex is applied for the written key. If the local mutex fails to be obtained, the transaction is rolled back. After the local mutex is successfully applied, the tuple consisting of the written key-value pair and the identifier of the current transaction is stored in the write set of the transaction. If the read operation cannot obtain the required local shared lock due to a local shared lock conflict, or if the write operation cannot obtain the required local mutex due to a local mutex conflict, the current transaction needs to be rolled back and all local mutexes and local shared locks applied by the current transaction are released. The write set is a set that records the write tuples of the transaction write operation; The tuple is the basic unit for storing data, and its content includes a key for indexing the tuple, representing the value and version of the user data; the tuple version is the identifier of the last transaction that modified the tuple; the transaction identifier is a 64-bit integer, and each transaction has a unique identifier.

Images