A method of transaction concurrency control

Abstract

The application discloses a transaction concurrent control method, first constructs a set of transaction system, each transaction index in the transaction system collects the transaction characteristic information of the data item of the partition in real time and dynamically adjusts the partition of the transaction, selects the appropriate concurrent control method for each partition, generates a concurrent control plan, and gives the partition manager, the partition manager adjusts the data partition in the transaction index according to the concurrent control plan, and distributes the concurrent control plan to the client, according to the partition range in the concurrent control plan and the concurrent control method of each partition, carries out the read-write operation, and completes the adaptive transaction concurrent control method. The method of the application adopts the adaptive concurrent control mechanism based on the transaction characteristics of the data item, dynamically selects the appropriate concurrent control method according to the transaction scale, the conflict rate and other characteristics, balances the load between the nodes, and effectively deals with the real-time variable transaction type and different workloads.

Description

Technical Field

[0001] This invention belongs to the field of distributed database technology, specifically relating to a transaction concurrency control method. Background Technology

[0002] A transaction is an ordered set of operations that access or modify data items, guaranteeing the ACID properties. The task of transaction concurrency control algorithms is to ensure that the isolation of transactions is not violated when multiple transactions access the same data in the database simultaneously. Mainstream transaction concurrency control algorithms include the following techniques:

[0003] Two-phase locking (2PL) ensures isolation by acquiring locks during transaction execution. In the growth phase, a transaction acquires shared locks for reading data items and exclusive locks for writing data items. A transaction may need to wait for a lock to be released if the requested data item is already locked. In the shrink phase, the transaction releases its locks and is not allowed to acquire new locks. Two-phase locking is advantageous in scenarios with a large number of data item conflicts, as it ensures the effective progress of the transaction through locking. However, in scenarios with few conflicts, the locking mechanism offers no advantage. Conversely, the overhead of locking (network communication overhead in a distributed environment) gradually becomes a bottleneck affecting concurrency control performance.

[0004] Optimistic Concurrency Control (OCC): During the execution phase, transactions are executed concurrently without interfering with each other. During the verification phase, the transaction's serializability is verified against all transactions committed since the beginning of the transaction or all transactions currently in the verification phase to determine the execution result. If the transaction can be committed, its modifications are persisted to storage; otherwise, the transaction is rolled back. Compared to two-phase commit, optimistic concurrency control saves unnecessary locking overhead in low-conflict scenarios. However, in high-conflict scenarios, while two-phase commit allows transactions to commit sequentially, optimistic concurrency control results in many transaction rollbacks, impacting transaction execution efficiency.

[0005] Partition Concurrency Control (PartCC) divides all data items into multiple data partitions, each with a unique lock. Modifications to any data item within a transaction require acquiring the lock on the partition containing that data item, and the lock is released at the end of the transaction. This concurrency control method is suitable for partitionable transaction scenarios, where all modified data items within a transaction reside in a single data partition, meaning the transaction only needs to acquire the lock for that partition. However, when all modified data items within a transaction are distributed across multiple different partitions, the performance of partition concurrency control degrades due to the need to acquire locks for multiple partitions.

[0006] Two-Phase Commit (2PC) divides distributed transactions into two phases: the prepare phase and the commit phase. In the prepare phase, the coordinator queries all participants to determine if the transaction can be executed. Only after all participants have confirmed their consent can the commit phase begin, allowing the participants to actually execute the transaction.

[0007] It's clear from the above concurrency control methods that a suitable method involves trade-offs. For different load scenarios, the throughput of the same concurrency control method can vary several times. Therefore, no single concurrency control method is suitable for all load scenarios, and an inappropriate method may lead to a significant performance degradation. Business personnel need to analyze and weigh the degree, type, and scale of transaction conflicts in the workload before making a selection. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention proposes a transaction concurrency control method to cope with real-time and ever-changing transaction types and different workloads. Based on the real-time collected data item transaction characteristic information (i.e., data item transaction characteristic information), the method adjusts the data partitioning and selects an appropriate concurrency control method for each partition (the corresponding concurrency control method is used for write operations on data in each partition). At the same time, based on the partition load information, the method moves or merges partitions to balance the load among nodes.

[0009] The technical solution adopted in this invention is: a transaction concurrency control method, the specific steps of which are as follows:

[0010] S1. Construct a transaction system;

[0011] S2. In the transaction system, each transaction index collects the transaction characteristic information of the data items in the partition in real time and dynamically adjusts the partition of the transaction.

[0012] S3. Based on step S2, select an appropriate concurrency control method for each partition, generate a concurrency control plan, and hand it over to the partition manager;

[0013] S4. The partition manager adjusts the data partitions in the transaction index according to the concurrency control plan and distributes the concurrency control plan to the client. Based on the partition range in the concurrency control plan and the concurrency control method of each partition, read and write operations are performed to complete the adaptive transaction concurrency control method.

[0014] Furthermore, in step S1, the transaction system includes a client SC, a transaction system TS, and a storage system SS. The TS includes a transaction controller TC and multiple transaction indexes TI.

[0015] Furthermore, in step S1, the transaction index TI is specifically as follows:

[0016] The TI includes: TxIndex, DERegion, and DEBucket.

[0017] Each TI provides a TxOpService service, which is implemented by TxOpServiceImpl; the TxIndex data structure in TxOpServiceImpl implements the methods of the TxOpService service.

[0018] TxIndex consists of two parts: RegionPartitionTable and PartitionManager.

[0019] Wherein, RegionPartitionTable is a map.<Range,DERegionPtr,RangeComparator> The type is a collection of data partitions consisting of a mapping from a data range to a region. When a method implemented by TxIndex is called, it first finds the corresponding DERegion from RegionPartitionTable based on the data range being called, and then continues to call the methods of TxOpService implemented by DERegion.

[0020] PartitionManager is a background module independent of RegionPartitionTable. It continuously calls the PartitionService to obtain the latest data partition information belonging to the TxIndex and adjusts the existing partitions in RegionPartitionTable.

[0021] DERegion represents a data set consisting of all data within a data range.

[0022] A DERegion contains a fixed number of DEBuckets, which are responsible for storing data items. Each data item in the data-range is hashed into a fixed DEBucket.

[0023] When a method of TxOpService implemented by DERegion is called, it first finds the DEBucket to which it belongs based on the data being called, and then continues to call the method of TxOpService implemented by DEBucket.

[0024] DEBucket consists of DEMVCC and a mutex.

[0025] When a method of TxOpService implemented by DEBucket is called, a mutex lock is first used to ensure atomic-read-and-write semantics during the transaction processing phase. DEMVCC stores multi-version data items.

[0026] Within DERegion, there are three backend modules: RegionPersist, RegionDependence, and RegionMetric. These modules are responsible for sinking the committed data in this region to SS, collecting and reporting dependencies, and collecting and reporting statistical information, respectively, as follows:

[0027] RegionPersist periodically calls the GetMinATS method of the RegionService service. Then, based on the latest min-ats, it iterates through the buckets of that region, looking for committed data that can be pushed into StorageService. Finally, it calls the BatchStore method of the StorageService service to push the data into StorageService and clears the data that has been pushed into DEBucket.

[0028] RegionMetric collects read and write statistics for its region over a period of time. These statistics use a bvar structure, specifically the bvar::LatencyRecorder data type, to record latency and TPS information, and are updated after each read / write operation involving data items in that region is completed.

[0029] Ultimately, RegionMetric periodically calls the RegionMetric method of the RegionService to report to TC. The statistics recorded in RegionMetric reflect the partition load in P (partition).

[0030] The DEMVCC structure record includes statistics on the number of write conflicts and the number of transaction operations, i.e., bvar::Window <Adder <int>Data type, record the cumulative value in a period of time, these information reflect the size of C(data) and L(data), and can be calculated from the D(data) of the data item. When the D(data) of the data item is greater than λ, join the hot data item set in the RegionMetric. Finally, it will be reported to the TC in the RegionMetric method of the RegionService service. The TC adjusts the distribution of data in the region according to the statistical information of each region, including splitting or merging the region, and finally the PartitionManager adjusts the data in the partition.

[0031] Further, in the step S1, the transaction controller TC is specifically as follows:

[0032] The TC includes: DependenceManager, CCPlanner, PartitionManager and TxIDTable.

[0033] The TC provides three services of TxService, RegionService and PartitionService, which are implemented by respective ServiceImpl. The ServiceImpl includes three modules of DependenceManager, CCPlanner and PartitionManager, which are responsible for the discovery of dependence conflict, the selection of concurrent control method, and the management and change of data partition, respectively.

[0034] TxIDTable is a structure for generating, storing and recycling the unique TxID of each transaction. The main information saved in the TxID includes the start / end time stamp of the transaction, the set of other TxIDs of dependence and dependence, and the callback function registered in the transaction start phase for advance verification.

[0035] Among them, TxID is stored and indexed by TxIDMap. TxIDMap is a sharded map<TimeStamp, TxIDPtr> structure, which is used to establish and store the mapping relationship from the transaction start time stamp to the TxID structure.

[0036] The RegionMetric method of the RegionService service will be called by each partition periodically, and the statistical information reported includes:

[0037] (1) In the last time period, the total number of read / write requests processed by the partition.

[0038] (2) In the last time period, all data items in the partition whose D(data) exceeds the threshold λ;

[0039] RegionServiceImpl then passes the received partition statistics to CCP1anner in turn. CCP1anner adjusts the partitions according to the received statistics of each partition in the following order:

[0040] 1. Record all the data items whose data (D) exceeds the threshold λ in the last time period in the partition, i.e. the pessimistic data items, in PartitionConfig. The SC will lock according to the pessimistic data items in PartitionConfig;

[0041] 2. If the number of all the pessimistic data items in the partition exceeds the threshold, perform partition splitting;

[0042] 3. If the total number of read and write requests processed by the partition in the last time period exceeds the threshold λ, perform partition splitting;

[0043] 4. If the partition can meet the above two conditions without splitting after being combined with the left partition, perform partition combining.

[0044] CCP1anner then passes the partition change information generated according to the partition statistics to PartitionManager.

[0045] PartitionManager saves the current partition routing table (map<Range, PartitionConfig, RangeComparator> data type). The SC and TI constantly obtain the latest partition routing table through the GetPartition method of the PartitionService service. When PartitionManager receives the partition change information from CCP1anner, it first applies the change information to the partition routing table and increments the version number of the partition routing table. After all the TIs obtain the updated partition routing table, the PartitionManager in the TI adjusts the DERegion in RegionPartitionTable. After all the TIs complete the update of the partition routing table, the next partition change is continued.

[0046] Further, the step S2 is specifically as follows:

[0047] First, set the transaction characteristics of the two types of data items:

[0048] Data Entry Contention C(data) represents the degree of competition of concurrent transactions on the data item, which is represented by the conflict rate of the data item in a period of time:

[0049]

[0050] Where W represents the number of write operations on the data item within a certain period, including write-ahead operations and locks, and w∈W represents the number of write conflicts in W; C(data) represents the conflict rate of the data item within a certain period, and the closer C(data) is to 1, the higher the conflict rate of the data item, and the more necessary it is to adopt a pessimistic locking concurrency control method for the data item to reduce the probability of conflict rollback; the closer C(data) is to 0, the lower the conflict rate of the data item, and the more suitable it is to adopt an optimistic concurrency control method to reduce the overhead of locking.

[0051] Data Entry Txn Length L(dara) represents the size of the transaction after the write operation on any data item.

[0052]

[0053]

[0054] Among them, O t O represents the total number of operations in transaction t. t ∈O t Let l(data,t) represent the number of operations in transaction t from the time the data item 'data' is written until the time the transaction commits. Let l(data,t) represent the ratio of the number of operations in transaction t from the time the data item 'data' is written until the time the transaction commits to the total number of operations in t. Let L(data) represent the weighted average of l(data,t) of all transactions involving the write operation of data item 'data' over a certain period of time, according to the transaction size. The larger l(data,t) is, the earlier the data item 'data' is written in transaction t, and the more necessary it is to use a pessimistic locking concurrency control method for that data item to prevent a large number of subsequent operations from needing to be rolled back due to conflicts in writing data items. Conversely, an optimistic concurrency control method is more suitable.

[0055] When selecting a concurrency control method for a given data item, both C(data) and L(data) characteristics are considered, and the Data Pessimism Degree D(data) is set as follows:

[0056] D(data)=α*C(data)+(1-α)*L(data)

[0057] Where α represents the system parameter, and represents the proportion of the two features C(data) and L(data) in D(data); when D(data) reaches the preset threshold λ, a pessimistic concurrency control strategy will be used for the data item, and when it is below the threshold λ, an optimistic concurrency control strategy will be adopted.

[0058] In this transaction system, global data items are partitioned according to data range, and all partitions are distributed across TI. The Partition Load P(partition) measures the load on a data partition and the degree of transaction contention; it consists of two parts:

[0059] (1) The total number of read and write operations in the partition over a period of time reflects the load size of the data partition;

[0060] (2) The number of data items in the partition whose D(data) is greater than λ within a certain period of time reflects the degree of transaction competition for data items in the partition.

[0061] Each transaction index collects and calculates the C(data), L(data), D(data), and P(partition) information within the partition in real time, as well as the size and range of the partition, and the corresponding concurrency control method for the partition is adjusted in real time based on the characteristic information within the partition.

[0062] Furthermore, step S3 is specifically as follows:

[0063] Based on step S2, each transaction index TI collects and calculates the C(data), L(data), D(data), and P(partition) information of all partitions belonging to it, and reports it to the concurrency strategy selector CCPlanner at regular intervals. CCPlanner dynamically adjusts the transaction partitions based on real-time characteristics and statistical information, and selects an appropriate concurrency control method for each partition, thereby generating a concurrency control plan CCPlan.

[0064] The beneficial effects of this invention are as follows: The method of this invention first constructs a transaction system. Within this system, each transaction index collects transaction characteristic information of data items in a partition in real time and dynamically adjusts the partitions of the transaction. For each partition, a suitable concurrency control method is selected, generating a concurrency control plan, which is then given to the partition manager. The partition manager adjusts the data partitions in the transaction index according to the concurrency control plan and distributes the concurrency control plan to the client. Read and write operations are performed based on the partition range in the concurrency control plan and the concurrency control method for each partition, thus completing an adaptive transaction concurrency control method. This invention, with its adaptive concurrency control mechanism based on data item transaction characteristics, dynamically selects a suitable concurrency control method according to characteristics such as transaction size and conflict rate, balancing the load between nodes and effectively coping with real-time, ever-changing transaction types and different workloads. Attached Figure Description

[0065] Figure 1 This is a flowchart of a transaction concurrency control method according to the present invention.

[0066] Figure 2 This is a structural framework diagram of the transaction system in an embodiment of the present invention.

[0067] Figure 3 This is a TI architecture diagram in an embodiment of the present invention.

[0068] Figure 4 This is a graph showing the statistical information recorded in RegionMetric in an embodiment of the present invention.

[0069] Figure 5 This is a graph showing the statistical information recorded in DEMVCC in an embodiment of the present invention.

[0070] Figure 6 This is a diagram of the TC architecture in an embodiment of the present invention.

[0071] Figure 7 This is a schematic diagram illustrating the dynamic changes in data partitioning in an embodiment of the present invention.

[0072] Figure 8 This is a schematic diagram of the concurrency control plan in an embodiment of the present invention. Detailed Implementation

[0073] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0074] like Figure 1 The flowchart of a transaction concurrency control method of the present invention is shown below, and the specific steps are as follows:

[0075] S1. Construct a transaction system;

[0076] S2. In the transaction system, each transaction index collects the transaction characteristic information of the data items in the partition in real time and dynamically adjusts the partition of the transaction.

[0077] S3. Based on step S2, select an appropriate concurrency control method for each partition, generate a concurrency control plan, and hand it over to the partition manager;

[0078] S4. The partition manager adjusts the data partitions in the transaction index according to the concurrency control plan and distributes the concurrency control plan to the client. Based on the partition range in the concurrency control plan and the concurrency control method of each partition, read and write operations are performed to complete the adaptive transaction concurrency control method.

[0079] In this embodiment, step S1 is specifically as follows:

[0080] like Figure 2 As shown, the transaction system includes: a client (SC, SDK Client), a transaction system (TS, Transaction System), and a storage system (SS, Storage System).

[0081] The SC exposes the data item transaction interface and is responsible for starting and ending transactions, as well as handling transaction requests. This includes caching requests and forwarding read and write requests to the SS or TS based on data sharding information.

[0082] SS is responsible for storing committed data in TS in multiple versions for SC to read. SS can be implemented by various storage services, including object storage and file storage. These storage services need to implement the API defined by SS to complete the interaction with SC and TS.

[0083] The Transaction Controller (TS) is responsible for concurrency control between transactions and caching committed data. The TS consists of a Transaction Controller (TC) and multiple Transaction Indices (TIs). Global data is partitioned according to data range and distributed across multiple TIs. Each TI is responsible for caching committed data and storing write-ahead or lock information for uncommitted transactions to provide read / write and conflict detection services, and periodically pushes committed data down to the Service Controller (SS). It is also responsible for collecting transaction characteristics and shard load information and reporting them to the TC. The TC, in turn, is responsible for timestamp allocation, data shard scheduling, and selecting concurrency control methods based on transaction characteristics.

[0084] In this embodiment, the transaction index TI in step S1 is specifically as follows:

[0085] The architecture of Transaction Index (TI) is as follows: Figure 3 As shown.

[0086] The TI includes: TxIndex, DERegion, and DEBucket.

[0087] Each TI provides a TxOpService service, which is implemented by TxOpServiceImpl. The TxIndex data structure in TxOpServiceImpl implements the methods of the TxOpService service.

[0088] TxIndex consists of two parts: RegionPartitionTable and PartitionManager.

[0089] RegionPartitionTable is a map<Range,DERegionPtr,RangeComparator> The type is a collection of data partitions consisting of a mapping from a data range to a region. When a method implemented by TxIndex is called, it first finds the corresponding DERegion from RegionPartitionTable based on the data range being called, and then continues to call the methods of TxOpService implemented by DERegion.

[0090] PartitionManager is a background module independent of RegionPartitionTable. It continuously calls the PartitionService to obtain the latest data partition information belonging to the TxIndex and adjusts the existing partitions in RegionPartitionTable.

[0091] A DERegion is a collection of all data within a data range. A DERegion contains a fixed number of DEBuckets, which store data items. Each piece of data within the data range is hashed into a specific DEBucket. When a method of the TxOpService implemented by the DERegion is called, the DEBucket to which the called data belongs is first located, and then the method of the TxOpService implemented by the DEBucket is invoked.

[0092] DEBucket consists of DEMVCC and a mutex.

[0093] When a method of TxOpService implemented by DEBucket is called, a mutex lock is first used to ensure atomic-read-and-write semantics during the transaction processing phase. DEMVCC stores multi-version data items.

[0094] Within DERegion, there are three backend modules: RegionPersist, RegionDependence, and RegionMetric. These modules are responsible for sinking the committed data in this region to SS, collecting and reporting dependencies, and collecting and reporting statistical information, respectively, as follows:

[0095] RegionPersist periodically calls the GetMinATS method of the RegionService service. Then, based on the latest min-ats, it iterates through the buckets of that region, looking for committed data that can be pushed into StorageService. Finally, it calls the BatchStore method of the StorageService service to push the data into StorageService and clears the data that has been pushed into DEBucket.

[0096] like Figure 4 As shown, RegionMetric collects read and write statistics for its region over a period of time. These statistics use a bvar structure (bvar::LatencyRecorder data type, recording latency and TPS information) and are updated after each read / write operation involving data items in that region is completed.

[0097] Ultimately, RegionMetric periodically calls the RegionMetric method of the RegionService to report to TC. The statistics recorded in RegionMetric reflect the partition load in P (partition).

[0098] In addition, such as Figure 5 As shown, the DEMVCC structure also records statistical information including the number of write conflicts and the number of transaction operations (bvar::Window). <Adder <int>Data Type, which records the accumulated value in a period of time), these information reflect the size of C(data) and L(data), and D(data) of the data item can be calculated. When D(data) of a data item is greater than λ, it will join the hot data item set in RegionMetric. Finally, it will be reported to TC in the RegionMetric method of RegionService service. TC will adjust the distribution of data in the region according to the statistical information of each region (including splitting or merging the region) and finally adjust the data in the partition by PartitionManager.

[0099] In the step S1, the transaction controller TC is specifically as follows:

[0100] The architecture of Transaction Controller (TC) is shown in Figure 6 .

[0101] The TC includes DependenceManager, CCP planner, PartitionManager and TxIDTable.

[0102] TC provides TxService, RegionService and PartitionService three services, which are implemented by the respective ServiceImpl. ServiceImpl includes DependenceManager, CCP planner and PartitionManager three modules, which are responsible for the discovery of dependence conflict, the selection of concurrent control method, the management and change of data partition respectively.

[0103] TxIDTable is a structure for generating, storing and recycling the unique TxID of each transaction. The main information saved in TxID includes the start / end timestamp of the transaction, the set of other TxID that the transaction depends on and is depended on, and the callback function registered in the transaction start phase for early validation.

[0104] TxID is stored and indexed by TxIDMap. TxIDMap is a sharded map<TimeStamp, TxIDPtr> structure, which is used to establish and store the mapping relationship from the transaction start timestamp to the TxID structure.

[0105] RegionMetric method of RegionService service will be called by each partition periodically, and the statistical information reported includes:

[0106] (1) The total number of read and write requests handled by the partition in the last time period.

[0107] (2) The number of data items in the partition whose D(data) exceeds threshold λ in the last time period.

[0108] The RegionServiceImpl then passes the received partition statistics to the CCP1anner in turn. The CCP1anner adjusts the partitions according to the statistics of each partition received in the following order:

[0109] 1. Record all data items whose D(data) exceeds threshold λ in the partition in the last time period (pessimistic data items) in the PartitionConfig. The SC will lock according to the pessimistic data items in the PartitionConfig;

[0110] 2. If the number of pessimistic data items in the partition exceeds the threshold at this time, split the partition;

[0111] 3. If the total number of read and write requests handled by the partition in the last time period exceeds threshold λ, split the partition;

[0112] 4. If the partition can meet the first two conditions not to be split after being merged with the partition to its left, merge the partition.

[0113] Partition splitting is to prevent too many pessimistic data items in a partition, which leads to a large PartitionConfig and a high load of a single partition. Partition merging is to reduce inactive partitions and save system resources. The CCP1anner then passes the partition change information generated according to the partition statistics to the PartitionManager.

[0114] The CCP1anner then passes the partition change information generated according to the partition statistics to the PartitionManager.

[0115] The current partition routing table (map<Range, PartitionConfig, RangeComparator> data type) is saved in the PartitionManager, and the SC and TI constantly acquire the latest partition routing table through the GetPartition method of the PartitionService service. When the PartitionManager receives the partition change information from the CCP planner, the change information is first applied to the partition routing table, and the version number of the partition routing table is incremented. After all TIs acquire the updated partition routing table, the PartitionManager in the TI adjusts the DERegion in the RegionPartitionTable. Only when all TIs have completed the update of the partition routing table, the next partition change can continue.

[0116] In the embodiment, the step S2 is specifically as follows:

[0117] First, the transaction characteristics of two data items are set:

[0118] Data Contention C(data) represents the degree of competition of concurrent transactions on data items, which is represented by the conflict rate of the data item in a period of time:

[0119]

[0120] Wherein, W represents the number of all write operations on the data item in a period of time, the write operation includes pre-write and lock, and w∈W represents the number of write conflicts in W; C(data) represents the conflict rate of the data item in a period of time, and the closer C(data) is to 1, the higher the conflict rate of the data item is, and the more need to use the pessimistic locking concurrent control method for the data item to reduce the probability of conflict rollback; the closer C(data) is to 0, the smaller the conflict rate of the data item is, and it is suitable to use the optimistic concurrent control method to reduce the overhead of locking.

[0121] Data Entry Txn Length L(data) represents the size of the transaction after writing the data item operation in the transaction involving writing operation on any data item:

[0122]

[0123]

[0124] Wherein, O t represents the total number of operations of transaction t, o t ∈O t l(data, t) represents the ratio of the number of operations after writing the data item data to the commit of the transaction t to the total number of operations of the transaction t; L(data) represents the weighted average of l(data, t) of all transactions involving operations on the data item data within a period of time according to the size of the transaction; the larger l(data, t) is, the earlier the data item data is in the transaction t, and the more necessary it is to use a pessimistic locking method for the concurrent control of the data item to prevent a large number of subsequent operations from being rolled back due to write data item conflicts; otherwise, an optimistic concurrent control method is suitable;

[0125] Since multiple transactions may simultaneously involve operations on the data item data, and the total number of operations of each transaction is different, both C(data) and L(data) need to be considered when selecting a concurrent control method for a data item data, and a Data Pessimism Degree D(data) is set:

[0126] D(data) = a * C(data) + (1-a) * L(data)

[0127] wherein a represents a system parameter, indicating the proportion of C(data) and L(data) in D(data); when D(data) reaches a pre-set threshold λ, a pessimistic concurrent control strategy is used for the data item, and when D(data) is lower than the threshold λ, an optimistic concurrent control strategy is used.

[0128] In the transaction system, global data items are partitioned according to data-range, and all partitions are dispersed in TIs. A Partition Load P(partition) is set to measure the load size and transaction competition degree of a data partition, which consists of two parts:

[0129] (1) the total number of read and write operations in the partition within a period of time, reflecting the load size of the data partition;

[0130] (2) the number of data items with D(data) greater than λ in the partition within a period of time, reflecting the transaction competition degree of the data items in the partition.

[0131] Each transaction index collects and calculates C(data), L(data), D(data), and P(partition) information in the partition in real time, the size and range of the partition, and the concurrent control method corresponding to the partition is adjusted in real time according to the characteristic information in the partition.

[0132] For example Figure 7 As shown in this embodiment, the process of dynamically adjusting data partitions due to real-time data hotspots is as follows:

[0133] 1. Initially, there are two partitions [-∞, data1) and [data1, +∞), both of which use the OCC concurrency control method.

[0134] 2. At a certain moment, a large number of transactions access data item data2, causing its D(data) to exceed the threshold λ, thus making it a pessimistic data item in the [data1, +∞) partition. The 2PL concurrency control method is then applied. The remaining data items within the partition continue to use the OCC concurrency control method.

[0135] 3. After a period of time, the D(data) values ​​of data3, data4, data5, and data6 within the [data1, +∞) partition also exceed the threshold λ. Therefore, these data items become pessimistic data items and require the use of the 2PL concurrency control method. Since the number of pessimistic data items in the [data1, +∞) partition is too large, exceeding the threshold μ, it needs to be split into two partitions: [data1, data4) and [data4, +∞).

[0136] 4. After a period of time, there are no more data items with D(data) greater than λ in the partitions [data2, data4) and [data4, +∞), so all data items in these two partitions will use the concurrency control method of OCC.

[0137] Ultimately, since the two partitions [data2, data4) and [data4, +∞) are adjacent and the number of all pessimistic data items in the partitions is less than the threshold μ, the two partitions were merged to reduce the number of partitions.

[0138] In this embodiment, step S3 is specifically as follows:

[0139] like Figure 1 As shown, based on step S2, each transaction index TI collects and calculates the C(data), L(data), D(data), and P(partition) information for all partitions belonging to it, and periodically reports this information to the concurrency strategy selector CCPlanner. CCPlanner dynamically adjusts the transaction partitions based on real-time characteristics and statistical information, and selects an appropriate concurrency control method for each partition, thereby generating a concurrency control plan CCPlan. CCPlan is as follows... Figure 8 As shown.

[0140] To sum up, the method of the application balances the load among nodes and effectively deals with real-time and variable transaction types and different workloads by using an adaptive concurrency control mechanism based on transaction characteristics of data items and dynamically selecting appropriate concurrency control methods according to transaction size, conflict rate and other characteristics.

[0141] Those skilled in the art will understand that the embodiments described herein are for the purpose of helping the reader understand the principles of the application and should be understood as not limiting the scope of protection of the application to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations according to the technical inspiration disclosed in the application without departing from the essence of the application, and these modifications and combinations are still within the scope of protection of the application.< / int> < / int>

Claims

1. A transaction concurrency control method, comprising the following steps: S1, constructing a transaction system; In the step S1, the transaction index TI is specifically as follows: The TI comprises: TxIndex, DERegion, DEBucket; Each TI provides TxOpService service and is specifically implemented by TxOpServiceImpl; The TxIndex data structure in TxOpServiceImpl implements the method of TxOpService service; TxIndex includes RegionPartitionTable and PartitionManager; RegionPartitionTable is a map<Range, DERegionPtr, RangeComparator> type, that is, a data partition set composed of a mapping from data-range to region; when the method implemented by TxIndex is called, first, according to the called data-range, find out the corresponding DERegion from RegionPartitionTable, and then continue to call the method of TxOpService implemented by DERegion; PartitionManager is a background module independent of RegionPartitionTable, which continuously calls PartitionService service to obtain the latest data partition information belonging to the TxIndex, and adjusts the existing partitions in RegionPartitionTable; DERegion represents a data set composed of all data items in a data-range; DERegion contains a fixed number of DEBucket, and DEBucket is responsible for storing data item data, and each data item in the data-range is fixedly hashed to a DEBucket; When the method of TxOpService implemented by DERegion is called, first, find the DEBucket to which the called data item belongs, and then continue to call the method of TxOpService implemented by DEBucket; DEBucket includes DEMVCC and a mutex; When the method of TxOpService implemented by DEBucket is called, it will first use mutex locking to ensure atomic-read-and-write semantics in the transaction processing stage; DEMVCC stores multiple versions of data item data; RegionPersist, RegionDependence, RegionMetric three background modules in DERegion are responsible for sinking the committed data in this region to SS, dependence collection and reporting, statistical information collection and reporting, as follows: RegionPersist will periodically call the GetMinATS method of RegionService service; then based on the latest min-ats, traverse the bucket of the region to find the committed data sunk to SS; finally call the BatchStore method in StorageService service to sink the data to SS, and clear the data in DEBucket that has been sunk; RegionMetric will collect the read-write statistical information of its region within a period of time; these statistical information uses bvar structure, i.e. bvar::LatencyRecorder data type, to record latency and tps information, and update after each read-write process involving the data item of the region is completed; Finally, RegionMetric will periodically call the RegionMetric method of the RegionService service to report to TC; these statistics recorded in RegionMetric reflect the partition load situation in the Region ​ The DEMVCC structure records include statistics of the number of write conflicts, the number of transaction operations, i.e., bvar::Window<Adder <int>Data type, the accumulated value recorded in a period of time, which reflects and the size of the data item, and calculates the ; when the of the data item is greater than , it is added to the hot data item set in the RegionMetric; finally, it is reported to the TC in the RegionMetric method of the RegionService service; the TC adjusts the distribution of data in the region according to the statistical information of each region, including splitting or merging the region, and finally the PartitionManager adjusts the data in the partition;< / int> S2, the transaction system collects the transaction characteristic information of the data item of each partition in real time and dynamically adjusts the partition of the transaction; S3, based on step S2, select a suitable concurrency control method for each partition, generate a concurrency control plan, and hand it over to the partition manager; S4, the partition manager adjusts the data partition in the transaction index according to the concurrency control plan, and distributes the concurrency control plan to the client, and performs read-write operation according to the partition range in the concurrency control plan and the concurrency control method of each partition, to complete the adaptive transaction concurrency control method.

2. The method of claim 1, wherein, In step S1, the transaction system includes a client SC, a transaction system TS and a storage system SS, TS includes a transaction controller TC and multiple transaction indexes TI.

3. The method for transaction concurrency control according to claim 1, wherein, In step S1, the transaction controller TC is as follows: The TC includes: DependenceManager, CCPlanner, PartitionManager and TxIDTable; TC provides three services TxService, RegionService and PartitionService, which are implemented by respective ServiceImpl; ServiceImpl includes DependenceManager, CCPlanner and PartitionManager three modules, which are responsible for dependence conflict discovery, concurrency control method selection, data partition management and change respectively; TxIDTable is a structure for generating, storing and recycling the unique TxID of each transaction; the main information saved in TxID includes the start / end timestamp of the transaction, the set of other TxIDs that depend on and are depended on, and the callback function registered for early verification in the transaction start phase; Wherein, TxID is stored and indexed by TxIDMap; TxIDMap is a shard map<TimeStamp, TxIDPtr> structure, which is used to establish and store the mapping relationship from the transaction start timestamp to TxID structure; RegionMetric method of RegionService service is called by each partition periodically, and the statistical information reported includes: (1) In the last time period, the total number of read and write requests processed by the partition; (2) all data items in the partition that exceed the threshold in the last time period exceeding the threshold data items; Then RegionServiceImpl gives the received partition statistical information to CCPlanner in turn; CCPlanner adjusts the partition according to the received statistical information of each partition in the following order: (1) All data in the partition from the previous time period Exceeding the threshold The data items, i.e. pessimistic data items, are recorded in PartitionConfig. SC will lock based on the pessimistic data items in PartitionConfig. (2) If the number of all pessimistic data items in the partition exceeds the threshold value at this time, the partition is split; (3) If the total number of read and write requests processed by the partition exceeds the threshold in the previous time period. If so, then partitioning will be performed; (4) If the partition and its left partition are merged to meet the first two conditions, the partition is merged; Then CCPlanner transmits the partition change information generated according to the partition statistical information to PartitionManager; PartitionManager saves the current partition routing table map<Range, PartitionConfig, RangeComparator> data type, and SC and TI constantly obtain the latest partition routing table through the GetPartition method of PartitionService service; when PartitionManager receives the partition change information from CCPlanner, it first applies the change information to the partition routing table, and makes the version number of the partition routing table increment; after all TIs obtain the updated partition routing table, the PartitionManager in TI adjusts the DERegion in RegionPartitionTable; after all TIs complete the update of the partition routing table, the next partition change is continued.

4. The method for transaction concurrency control according to claim 1, wherein, The step S2 is specifically as follows: First, set the transaction characteristics of the two kinds of data items: Data Entry Contention Data Entry Contention represents the degree of contention of concurrent transactions on a data item, represented by the conflict rate of that data item over a period of time: ； wherein, represents the number of all write operations on the data item in a period of time, the write operations including pre-write and lock, ∈ represents the number of write conflicts in the data item; represents the conflict rate of the data item in a period of time, and the closer to 1, the higher the conflict rate of the data item, the more need to use the pessimistic locking concurrent control method for the data item to reduce the probability of conflict rollback; the closer to 0, the smaller the conflict rate of the data item, and it is suitable to use the optimistic concurrent control method to reduce the overhead of locking; Data Entry Txn Length represents the size of the transaction after the write operation to the data entry has been performed. ； ； wherein, the total number of operations of a transaction , ∈ the total number of operations of a transaction , the total number of operations of a transaction , the ratio of the total number of operations of a transaction to the total number of operations of a transaction , the larger the value, the more likely the write data item data is in the front of a transaction , and the more likely the pessimistic concurrency control method is needed for the data item to prevent a large number of subsequent operations from being rolled back due to write data item conflicts; otherwise, the optimistic concurrency control method is suitable. In making the concurrent control method selection for a certain data item data, both of the two features are considered simultaneously and The Data Pessimism Degree is set : ； wherein, represents a system parameter, represents and the proportion of two features in ; when a pre-set threshold is reached , a pessimistic concurrency control strategy will be used for the data item, and below the threshold , an optimistic concurrency control strategy is adopted; In the transaction system, the global data items are partitioned by data-range, and all the partitions are scattered in TIs. Partition Load The load size of a data partition and the transaction competition degree are measured by two parts: (1) The total number of read and write operations in a period of time, which reflects the load size of the data partition; (2) the number of data items in the partition within a period of time, reflecting the degree of transaction competition of data items in the partition; greater than the number of data items in the partition within a period of time, reflecting the degree of transaction competition of data items in the partition; Each transaction index collects and calculates the partition's , , , information in real time, the size of the partition, the range, and the concurrent control method corresponding to the partition is adjusted in real time according to the characteristic information in the partition.

5. The method for transaction concurrency control according to claim 1, wherein, The step S3 is specifically as follows: Based on step S2, each transaction index TI collects and calculates the respective , , , information of all partitions belonging to it and reports it to the concurrency control planner CCP planner in a timely manner; the CCP planner dynamically adjusts the partitioning of the transaction according to the real-time characteristics and statistical information, and selects a suitable concurrency control method for each partition, thereby generating a concurrency control plan CCP plan.

Images