System and method for supporting multi-version concurrency control in column storage database

Abstract

The invention discloses a system and a method for supporting multi-version concurrency control in a column storage database, belongs to the technical field of databases, and aims to solve the technical problem of how to provide efficient and flexible concurrency control and data operation management. Comprising a multi-version management module for creating a transaction ID and a timestamp for each transaction and maintaining a version linked list for each data segment in a column storage database; the operation optimization module is used for performing data operation on the data in the column storage database based on the submitted transaction, performing operation according to a comparison result of a version number and a transaction timestamp when the data operation is executed, and storing the data before modification into a revocation buffer area; the transaction conflict processing module is used for detecting and processing transaction conflicts based on a preset conflict processing rule; the storage space management module divides the storage of the column storage database into table storage and local storage; and the version linked list management module is used for deleting the version information which is no longer referenced by any active transaction based on the transaction state and the version information.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a system and method for supporting multi-version concurrency control in a column-stored database. Background Art

[0002] With the explosive growth of data volumes and the increasing complexity of data analysis requirements, column-based databases have gained widespread application in data warehousing, business intelligence, and other fields due to their significant advantages in data compression and query performance. In scenarios where multiple users concurrently access column-based databases, ensuring data consistency and efficient concurrent transaction execution becomes a key issue.

[0003] During concurrent operations, data read and write operations by different transactions may interfere with each other, leading to data inconsistencies such as dirty reads, non-repeatable reads, and phantom reads. Furthermore, while locking mechanisms can ensure data consistency to a certain extent, they incur high overhead, resulting in reduced concurrency performance and slower system responsiveness. Furthermore, efficient management of data versions and storage space during operations such as data updates, deletions, and insertions is another challenge facing column-based databases.

[0004] Existing technologies have many shortcomings in dealing with the above problems. The simple locking mechanism adopted by some database systems cannot meet the performance requirements in high-concurrency scenarios, and some complex concurrency control schemes increase the complexity and implementation difficulty of the system.

[0005] How to provide efficient and flexible concurrency control and data operation management to improve the overall performance and reliability of column-based databases is a technical problem that needs to be solved. Summary of the Invention

[0006] The technical task of the present invention is to address the above shortcomings and provide a system and method for supporting multi-version concurrency control in a column-based database to solve the technical problem of how to provide efficient and flexible concurrency control and data operation management to improve the overall performance and reliability of the column-based database.

[0007] In a first aspect, the present invention provides a system for supporting multi-version concurrency control in a column-stored database, comprising a multi-version management module, an operation optimization module, a transaction conflict handling module, a storage space management module, and a version linked list management module;

[0008] The multi-version management module is configured to perform the following: create a transaction ID and timestamp for each transaction, maintain a version linked list for each data segment in the column-based database, and store version information through nodes in the version linked list. The version information includes the version number and operation type when the transaction is used to operate on the data. When the transaction is used to operate on the data, before the transaction is committed, the version number is the transaction ID. When the transaction is committed, the version number is updated to the transaction timestamp.

[0009] The operation optimization module is used to perform the following operations: performing data operations on the data in the column-based database based on the submitted transactions, performing the operations according to the comparison results between the version number and the transaction timestamp, and storing the data before the modification in the undo buffer, wherein the data operations include scan operations, insert operations, delete operations, and update operations;

[0010] The transaction conflict handling module is used to perform the following: when operating on data based on a transaction, detecting and handling transaction conflicts based on preset conflict handling rules;

[0011] The storage space management module is used to perform the following operations: divide the storage of the column-based database into table storage and local storage, cache relevant data in the local storage before the transaction is committed, and merge the locally stored data into the table for persistence after the transaction is committed;

[0012] The version chain table management module is used to perform the following: scan the version information in the version chain table through a version cleanup task, and delete the version information that is no longer referenced by any active transaction based on the transaction status and version information.

[0013] Preferably, when performing a scanning operation, the operation optimization module follows the following rule: when the version number is greater than the timestamp of the current transaction and the version number is not equal to the transaction ID of the current transaction, the data corresponding to the version before the transaction is modified is selected.

[0014] Preferably, when performing a modification operation, the data in the original data segment is not modified. A dummy node is reserved at the head of the version linked list to save the updated data. When the data is changed, the modified value is saved directly in the dummy node at the head of the linked list, and the data before the change is saved in the undo buffer. At the same time, the pointer to the buffer and the version information are inserted into the head of the version linked list.

[0015] As a preference, in the distributed database KaiwuDB, each library table corresponds to two parts of storage: table storage and local storage;

[0016] When performing an insert operation, the operation optimization module is used to perform the following operations:

[0017] For the data to be inserted, the row group to be added, the columns in the row group, and the data segments in the columns are found. Based on the compression method used by the data segment, the corresponding compression algorithm is called to add the data to the data segment.

[0018] When adding data to a data segment, if the data segment space is insufficient, a temporary segment is created. When a row group is full, the data segment is flushed to disk and marked as added to the table. During a rollback, the area on disk storing data is marked as unused.

[0019] When a transaction is committed, storage commit, undo buffer commit, and WAL flush operations are performed.

[0020] Preferably, when performing a delete operation, the operation optimization module is configured to perform the following operations:

[0021] For the data to be deleted, determine whether the data is stored in the transaction local or in the table by checking whether the row number of the data is greater than MAX_ROW_ID;

[0022] When deleting data, the version information of the corresponding data is marked as the transaction ID of the current transaction, and a corresponding entry is added to the undo buffer to record the deleted row number;

[0023] When a transaction is committed, storage commit, undo buffer commit, and WAL flush operations are performed, and the version information in the table is updated to the transaction timestamp.

[0024] Preferably, when performing a modification operation, the operation optimization module is configured to perform the following operations:

[0025] For the data to be modified, determine the data location by its row ID and find the corresponding row group and column data;

[0026] When modifying the data to be modified, save the data before modification to the undo buffer, modify the data locally, and insert the modified information of the data into the corresponding version list;

[0027] When a transaction is committed, storage commit, undo buffer commit, and WAL flush operations are performed.

[0028] Preferably, when performing a scanning operation, the operation optimization module is configured to perform the following operations:

[0029] For the data to be scanned, first scan the table, then scan the local storage. When scanning the row group, compare the version numbers corresponding to the insert operation and the version numbers corresponding to the delete operation with the timestamp of the current transaction to determine which data is visible to the current transaction. If the version number corresponding to the data when the insert operation is performed is earlier than the timestamp of the current transaction, and the version number corresponding to the data when the delete operation is performed is later than the timestamp of the current transaction, or the data has not been deleted, then all the data is considered visible.

[0030] If all data is visible and there are no filtering conditions, directly read the data in each column. If all data is visible but there are filtering conditions, filter according to the filtering conditions and then read the corresponding column data.

[0031] Preferably, the transaction conflict handling module is used to perform the following operations: before modifying the data based on the current transaction, detect whether other transactions are modifying the data; if so, according to a preset strategy, the current transaction waits for a predetermined time and then attempts to modify the data again; if other transactions complete the reading operation on the data after waiting for the predetermined time, the data is modified based on the current transaction; if the data cannot be modified based on the current transaction after waiting for the predetermined time, the current transaction is rolled back.

[0032] In a second aspect, the present invention provides a method for supporting multi-version concurrency control in a column-based database, comprising the following steps:

[0033] Multi-version management: A transaction ID and timestamp are created for each transaction. A version list is maintained for each data segment in the column-based database. Version information is stored in nodes in the version list. This information includes the version number and operation type for transactions performed on data. Before a transaction is committed, the version number is the transaction ID. When the transaction is committed, the version number is updated to the transaction timestamp.

[0034] Operation optimization: Data operations are performed on data in the column-based database based on committed transactions. Data operations are performed based on the comparison results between the version number and the transaction timestamp, and the data before modification is stored in the undo buffer. Data operations include scan operations, insert operations, delete operations, and update operations.

[0035] Transaction conflict handling: When operating on data based on transactions, transaction conflicts are detected and handled based on preset conflict handling rules;

[0036] Storage space management: The column-based database is divided into table storage and local storage. Before a transaction is committed, relevant data is cached in the local storage. After the transaction is committed, the local storage data is merged into the table for persistence.

[0037] Version list management: Scan the version information in the version list through the version cleanup task, and delete the version information that is no longer referenced by any active transaction based on the transaction status and version information.

[0038] The system and method for supporting multi-version concurrency control in a column-based database of the present invention have the following advantages:

[0039] 1. Improve concurrency performance: Through the multi-version mechanism, different transactions can be executed concurrently, reducing the use of locks, lowering the performance overhead caused by lock contention, and improving the system's concurrent processing capabilities;

[0040] 2. Ensure data consistency: Utilize version management and transaction isolation mechanisms to ensure that each transaction sees a consistent view of data during concurrent operations, avoiding data inconsistencies such as dirty reads, non-repeatable reads, and phantom reads.

[0041] 3. Optimize transaction processing efficiency: Simplify the transaction commit and rollback process, perform transaction operations in local storage, merge them into the table when committing, and rollback at almost no cost, thereby improving the efficiency and reliability of transaction processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] The present invention will be further described below with reference to the accompanying drawings.

[0044] Figure 1 This is a schematic diagram of the structure of a version linked list in a system supporting multi-version concurrent control in a column-stored database according to Example 1;

[0045] Figure 2 This is a flowchart of a method for supporting multi-version concurrency control in a column-based database according to Example 2. DETAILED DESCRIPTION

[0046] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments given are not intended to limit the present invention. Unless there is a conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0047] Embodiments of the present invention provide a system and method for supporting multi-version concurrency control in a column-based database, which are used to solve the technical problem of how to provide efficient and flexible concurrency control and data operation management to improve the overall performance and reliability of the column-based database.

[0048] Example 1:

[0049] The present invention provides a system supporting multi-version concurrent control in a column-stored database, comprising a multi-version management module, an operation optimization module, a transaction conflict processing module, a storage space management module and a version linked list management module.

[0050] The multi-version management module is used to perform the following: create a transaction ID and timestamp for each transaction, maintain a version list for each data segment in the column-based database, and store version information through nodes in the version list. The version information includes the version number and operation type when operating on data based on the transaction. When operating on data based on the transaction, before the transaction is committed, the version number is the transaction ID. When the transaction is committed, the version number is updated to the transaction timestamp.

[0051] As a specific implementation of the multi-version management module, this module is used to confirm the transaction identifier and establish a version list.

[0052] Transaction identification is determined, and each newly created transaction is assigned a unique transaction ID (incrementing from 2^62) and a timestamp (incrementing from 2). While the transaction is uncommitted, the transaction ID is used as a temporary version number. After committing, the version number is updated to the timestamp. Each transaction can only see data where "version number < its own transaction timestamp" or "version number = its own transaction ID." This ensures isolation, as changes made by an uncommitted transaction are not visible to other transactions.

[0053] For example, suppose transactions Txn1, Txn2, and Txn3 are started sequentially. Txn1's transaction ID is 2^62, with a timestamp of 2; Txn2's transaction ID is 2^62+1, with a timestamp of 3; and Txn3's transaction ID is 2^62+2, with a timestamp of 4. When Txn1 modifies data but does not commit, other transactions (such as Txn2 and Txn3) will not see Txn1's modifications (because Txn1's version number is greater than the timestamps of Txn2 and Txn3). Only after Txn1 commits will its modifications become visible to other transactions.

[0054] A version linked list is established and maintained for each data segment (each column of data is split into multiple data segments according to a fixed number of rows) in the column-based database. The linked list nodes store version information. The version number in the version information is initialized to the transaction ID and updated to the transaction timestamp when the transaction is committed. The data before the transaction change is also recorded. During data scanning, the version number is compared with the timestamp of the current transaction to determine whether to apply the version data. The specific rule is: when the version number is greater than the timestamp of the current transaction and is not equal to the transaction ID of the current transaction, the version before the transaction modification is applied.

[0055] For example, a data segment stores employee salary information. The initial version number is 2, and the salary data is 5000. Transaction Txn4 (transaction ID 2^62+3, timestamp 5) modifies this data segment, changing the salary to 6000. During the modification, a new version node is created with an initial version number of 2^62+3 and the pre-modified data of 5000, and is inserted at the head of the version list. When transaction Txn5 (timestamp 6) scans this data segment, because Txn4's version number 2^62+3 is greater than Txn5's timestamp 6, and 2^62+3 is not equal to Txn5's transaction ID, Txn5 applies the pre-modified version of Txn4, resulting in the salary data being 5000.

[0056] This embodiment can provide multi-version concurrency control through this module. The granularity of multi-version concurrency control is for Segment, that is, part of the data in each column. Each Segment will maintain a version information (version info), including: insert version number, delete version number and modification version. The modification version maintains a version list. Figure 1 As shown in the figure, to support column compression, data changes do not modify the original segment data. Instead, a dummy node is reserved at the head of the version list to store the updated data. When data changes, the modified value is saved directly in the dummy node at the head of the list. The data before the change is saved in the undo buffer, and the buffer pointer and version information are inserted into the head of the version list. This preserves historical data versions for easy use during transaction rollback or data recovery.

[0057] For example, continuing with the employee salary example above, if Txn6 (transaction ID is 2^62+4, start time is 7) changes the salary from 6000 to 7000 again, it will modify the salary data of the dummy node to 7000, save 6000 in the undo buffer, and insert a new node at the head of the version list. The initial version number is 2^62+4, and the data before the change is 6000.

[0058] The operation optimization module is used to perform the following operations: perform data operations on the data in the column-stored database based on the submitted transactions, perform operations based on the comparison results of the version number and the transaction timestamp when performing data operations, and store the data before modification in the undo buffer. Among them, data operations include scan operations, insert operations, delete operations and update operations.

[0059] As a specific implementation of the operation optimization module, when executing an operation, the operation optimization module follows the following rules: when the version number is greater than the timestamp of the current transaction and the version number is not equal to the transaction ID of the current transaction, the data corresponding to the version before the transaction modification is selected.

[0060] In the distributed database KaiwuDB, each table has two storage components: table storage and local storage. When performing an insert operation, the operation optimization module performs the following operations: For the data to be inserted, it finds the row group to be added, the columns in the row group, and the data segments in the columns. Based on the compression method used by the data segment, it calls the corresponding compression algorithm to add the data to the data segment. When adding data to the data segment, if the data segment space is insufficient, a temporary segment is created. When a row group is full, the data segment is flushed to disk and marked as added to the table. During a rollback, the area on disk storing data is marked as unused. When the transaction is committed, it performs storage commit, undo buffer commit, and WAL flush operations.

[0061] Suppose there is an employee information table containing employee ID, name, and salary columns. Transaction Txn7 inserts a new employee record (employee ID 1001, name "Bob", and salary 5000). First, the system finds the corresponding row group, column, and segment. Assuming that the segment uses a compression algorithm (such as LZ4 compression), the data is compressed and added to the segment. If the segment does not have enough space, a temporary segment is created to store the data. When the row group is full, it is flushed to disk. When the transaction commits, the inserted record in the local storage is added to the table, the insertion information (such as employee ID, name, salary, and the timestamp of the insertion transaction) is written to the undo buffer log, and the corresponding version information in the table is updated. Finally, the WAL is flushed to disk.

[0062] When executing a delete operation, the operation optimization module is used to perform the following operations: for the data to be deleted, whether the data is stored locally in the transaction or in the table is determined by whether the row number of the data is greater than MAX_ROW_ID; when deleting data, the version information of the corresponding data is marked with the transaction ID of the current transaction, and a corresponding entry is added to the undo buffer to record the deleted row number; when the transaction is committed, the storage commit, undo buffer commit, and WAL disk flush operations are performed, and the version information in the table is updated to the transaction timestamp.

[0063] For example, in the employee information table described above, transaction Txn8 deletes the record with employee ID 1001. Because the row number of this record (assuming it's less than MAX_ROW_ID) indicates that it belongs to the table, the system marks the deletion ID of this record as Txn8's transaction ID and records the deleted row number in the undo buffer. When the transaction commits, the undo buffer commit writes the deleted row number to the log. A WAL flush ensures the operation is persistent, and the deleted version information in the table is updated with the transaction timestamp.

[0064] When performing a modification operation, the data in the original data segment is not modified. A dummy node is reserved at the head of the version list to store the updated data. When the data changes, the modified value is directly saved in the dummy node at the head of the list, and the data before the change is saved in the undo buffer. At the same time, the buffer pointer and version information are inserted into the head of the version list. This operation optimization module is used to perform the following operations to perform modification operations: for the data to be modified, the data location is determined by the row ID of the data, and the corresponding row group and column data are searched; when modifying the data to be modified, the data before the modification is saved in the undo buffer, the data is modified locally, and the data modification information is inserted into the corresponding version list; when the transaction is committed, the storage commit, undo buffer commit, and WAL disk flush operations are performed.

[0065] For example, suppose transaction Txn9 updates the salary of employee ID 1001 from 5000 to 5500. First, determine the location of the record and find the corresponding row group and column data. The original salary of 5000 is saved to the undo buffer, then updated locally to 5500, and the updated information is inserted into the update version chain. When the transaction commits, the storage commit, the undo buffer commit (writing the salary update information to the WAL, with the update version number set to the transaction timestamp), and the WAL flush are completed.

[0066] When performing a scan operation, the operation optimization module is used to perform the following operations: for the data to be scanned, first scan the table and then scan the local storage. When scanning the row group, determine which data is visible to the current transaction by comparing the version number corresponding to the insert operation and the version number corresponding to the delete operation with the timestamp of the current transaction. If the version number corresponding to the insert operation on the data is earlier than the timestamp of the current transaction, and the version number corresponding to the delete operation on the data is later than the timestamp of the current transaction or the data has not been deleted, then it is determined that all the data is visible; if all the data is visible and there is no filtering condition, directly read the data in each column; if all the data is visible but there is a filtering condition, filter according to the filtering condition and read the corresponding column data.

[0067] For example, transaction Txn10 scans the employee information table. When scanning the row group, the system compares the insert and delete versions with the timestamp of Txn10. If the record with employee ID 1001 has an insert version earlier than the timestamp of Txn10 and a delete version later than or without a delete version (i.e., it has not been deleted), then the record is visible to Txn10. If there are filtering conditions (such as only querying employees with a salary greater than 5000), the conditions are filtered first, and then the records that meet the conditions are read. For columns that have been updated (such as the salary column), according to the multi-version rules, if an updated version exists and meets the visibility rules, the updated salary data is applied.

[0068] The transaction conflict handling module is used to perform the following: when operating on data based on a transaction, detect and handle transaction conflicts based on preset conflict handling rules.

[0069] As a specific implementation of the transaction conflict processing module, this module is used to perform the following operations: before modifying the data based on the current transaction, detect whether other transactions are modifying the data; if so, according to the preset strategy, the current transaction waits for a predetermined time and then attempts to modify the data again; if other transactions complete the read operation on the data after waiting for the predetermined time, the data is modified based on the current transaction; if the data cannot be modified based on the current transaction after waiting for the predetermined time, the current transaction is rolled back.

[0070] Suppose transaction Txn11 is updating the salary data for employee ID 1001. Before the update, the system detects that transaction Txn12 is reading the same employee's salary data. At this point, according to a preset policy, Txn11 can wait for a period of time (e.g., 500 milliseconds) before retrying the update. If Txn12 completes the read operation during the wait, Txn11 can continue the update. If the wait timeout expires and the update is still unsuccessful, Txn11 rolls back the transaction to ensure data consistency.

[0071] The storage space management module is used to perform the following operations: divide the storage of the column-based database into table storage and local storage, cache relevant data in local storage before transaction submission, and merge the locally stored data into the table for persistence after transaction submission.

[0072] This embodiment implements local storage and merging in storage space management: database storage is divided into two parts: tables (representing disk state) and local storage (representing table operations within a transaction). Local storage is merged with tables only after a transaction is committed. This increases transaction concurrency and provides virtually no cost for rollbacks.

[0073] For example, multiple transactions operate simultaneously on an employee information table. Transaction Txn13 inserts a new record in local storage, while transaction Txn14 updates a record in local storage. Due to the isolation of local storage, these operations do not interfere with each other. When the transactions commit, Txn13 and Txn14 merge the changes from local storage into the table, ensuring data persistence. If Txn13 needs to be rolled back, the insert operation in local storage is simply abandoned, incurring almost no additional cost.

[0074] The version list management module is used to perform the following: scan the version information in the version list through the version cleanup task, and delete the version information that is no longer referenced by any active transaction based on the transaction status and version information.

[0075] When the system load is low, the version list management module initiates a version cleanup task, scanning version information and, based on the transaction status table and version information, deleting all versions no longer referenced by any active transactions. This scan is performed incrementally, scanning only a portion of the data at a time to avoid the impact of scanning large amounts of data at once on the column engine's performance. Version cleanup is performed asynchronously to avoid interruptions to the column engine's normal data processing.

[0076] The system of this embodiment solves the data consistency, performance and transaction processing efficiency issues of column-based databases in a concurrent environment by improving the multi-version concurrency control mechanism, optimizing the add, delete, modify and query operation processes, and managing data versions and storage space.

[0077] Example 2:

[0078] The present invention provides a method for supporting multi-version concurrency control in a column-stored database, which is implemented by the system disclosed in Example 1. The method includes five steps: multi-version management, operation optimization, transaction conflict handling, storage space management, and version list management.

[0079] Step S100 Multi-version management: Create a transaction ID and timestamp for each transaction, maintain a version list for each data segment in the column-based database, and store version information through nodes in the version list. The version information includes the version number and operation type when operating on data based on the transaction. When operating on data based on the transaction, before the transaction is submitted, the version number is the transaction ID. When the transaction is submitted, the version number is updated to the timestamp of the transaction.

[0080] This embodiment provides confirmation transaction identification and establishes a version chain list through multi-version management.

[0081] Transaction identification is determined, and each newly created transaction is assigned a unique transaction ID (incrementing from 2^62) and a timestamp (incrementing from 2). While the transaction is uncommitted, the transaction ID is used as a temporary version number. After committing, the version number is updated to the timestamp. Each transaction can only see data where "version number < its own transaction timestamp" or "version number = its own transaction ID." This ensures isolation, as changes made by an uncommitted transaction are not visible to other transactions.

[0082] A version linked list is established and maintained for each data segment (each column of data is split into multiple data segments according to a fixed number of rows) in the column-based database. The linked list nodes store version information. The version number in the version information is initialized to the transaction ID and updated to the transaction timestamp when the transaction is committed. The data before the transaction change is also recorded. During data scanning, the version number is compared with the timestamp of the current transaction to determine whether to apply the version data. The specific rule is: when the version number is greater than the timestamp of the current transaction and is not equal to the transaction ID of the current transaction, the version before the transaction modification is applied.

[0083] Step S200 operation optimization: perform data operations on the data in the column-stored database based on the submitted transactions, perform operations based on the comparison results of the version number and the transaction timestamp when executing data operations, and store the data before modification in the undo buffer, where data operations include scan operations, insert operations, delete operations, and update operations.

[0084] As a specific implementation of operation optimization, when executing an operation, the operation optimization module follows the following rules: when the version number is greater than the timestamp of the current transaction and the version number is not equal to the transaction ID of the current transaction, the data corresponding to the version before the transaction modification is selected.

[0085] In the distributed database KaiwuDB, each table has two storage components: table storage and local storage. When performing an insert operation, the following operations are performed: for the data to be inserted, the row group to be added, the columns in the row group, and the data segments in the columns are found. Based on the compression method used by the data segment, the corresponding compression algorithm is called to add the data to the data segment. When adding data to the data segment, if the data segment space is insufficient, a temporary segment is created. When a row group is full, the data segment is flushed to disk and marked as added to the table. During a rollback, the area on disk storing data is marked as unused. When the transaction is committed, the storage commit, undo buffer commit, and WAL flush operations are performed.

[0086] When executing a delete operation, the following operations are performed: for the data to be deleted, whether the data is stored locally in the transaction or in the table is determined by whether the row number of the data is greater than MAX_ROW_ID; when deleting data, the version information of the corresponding data is marked with the transaction ID of the current transaction, and a corresponding entry is added to the undo buffer to record the deleted row number; when the transaction is committed, the storage commit, undo buffer commit, and WAL disk flush operations are performed, and the version information in the table is updated to the transaction timestamp.

[0087] When performing a modification operation, the data in the original data segment is not modified. A dummy node is reserved at the head of the version list to store the updated data. When the data changes, the modified value is directly saved in the dummy node at the head of the list, and the data before the change is saved in the undo buffer. At the same time, the buffer pointer and version information are inserted into the head of the version list. This operation optimization module is used to perform the following operations to perform modification operations: for the data to be modified, the data location is determined by the row ID of the data, and the corresponding row group and column data are searched; when modifying the data to be modified, the data before the modification is saved in the undo buffer, the data is modified locally, and the data modification information is inserted into the corresponding version list; when the transaction is committed, the storage commit, undo buffer commit, and WAL disk flush operations are performed.

[0088] When performing a scan operation, the following operations are performed: for the data to be scanned, the table is scanned first, and then the local storage is scanned. When scanning the row group, the version number corresponding to the insert operation and the version number corresponding to the delete operation are compared with the timestamp of the current transaction to determine which data is visible to the current transaction. Among them, if the version number corresponding to the insert operation on the data is earlier than the timestamp of the current transaction, and the version number corresponding to the delete operation on the data is later than the timestamp of the current transaction or the data has not been deleted, it is determined that all the data is visible; if all the data is visible and there is no filtering condition, the data of each column is read directly. If all the data is visible but there is a filtering condition, the data of the corresponding column is read after filtering according to the filtering condition.

[0089] Step S300: Transaction conflict processing: When operating on data based on a transaction, transaction conflicts are detected and processed based on preset conflict processing rules.

[0090] As a specific implementation of transaction conflict processing, the following operations are performed: before modifying the data based on the current transaction, detect whether other transactions are modifying the data. If so, according to the preset strategy, the current transaction waits for a predetermined time and then attempts to modify the data again. If other transactions complete the read operation on the data after waiting for the predetermined time, the data is modified based on the current transaction. If the data cannot be modified based on the current transaction after waiting for the predetermined time, the current transaction is rolled back.

[0091] Step S400: Storage space management: The storage of the column-based database is divided into table storage and local storage. Before a transaction is committed, relevant data is cached in the local storage. After the transaction is committed, the locally stored data is merged into the table for persistence.

[0092] This embodiment implements local storage and merging in storage space management: database storage is divided into two parts: tables (representing disk state) and local storage (representing table operations within a transaction). Local storage is merged with tables only after a transaction is committed. This increases transaction concurrency and provides virtually no cost for rollbacks.

[0093] Step S500: Version list management: Scan the version information in the version list through the version cleanup task, and delete the version information that is no longer referenced by any active transaction based on the transaction status and version information.

[0094] When the system load is low, the version list manager initiates a version cleanup task, scanning version information and, based on the transaction status table and version information, deleting all versions no longer referenced by any active transactions. This scan is performed incrementally, scanning only a portion of the data at a time to avoid impacting the column engine's performance by scanning large amounts of data at once. Version cleanup is performed asynchronously to avoid interruptions to the column engine's normal data processing.

[0095] The method disclosed in this embodiment first assigns each transaction a unique transaction ID (incrementing from 2^62) and a timestamp that increments from 2. The transaction ID is used as a temporary version number when uncommitted, and is updated to the timestamp after committing to ensure isolation. A version list is maintained for each data segment, and the version number and transaction timestamp are compared according to rules to determine data visibility. Version information is then stored in a specific manner. During addition, modification, deletion, and query operations, data operations and transaction submission are performed according to an optimized process. Insert operations create temporary segments based on space availability. The deletion operation module marks deleted data and records it. The update operation module saves old data, modifies local data, and updates the version chain. The scan operation module reads data according to multi-version rules and filtering conditions. In addition, system performance and data consistency are guaranteed by detecting and handling conflicts, managing storage space (merging local storage with the table after commit), and organizing the version list (asynchronous incremental scanning to delete useless versions under low load). By improving the multi-version concurrency control mechanism, optimizing the addition, deletion, modification, and query operation process, and managing data versions and storage space, the data consistency, performance, and transaction processing efficiency issues of column-based databases in concurrent environments are solved.

[0096] The above is a detailed introduction to the system and method for supporting multi-version concurrency control in a column-based database provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A system supporting multi-version concurrency control in a column-based database, characterized in that: It includes multi-version management module, operation optimization module, transaction conflict handling module, storage space management module and version list management module; The multi-version management module is configured to perform the following: create a transaction ID and timestamp for each transaction, maintain a version linked list for each data segment in the column-based database, and store version information through nodes in the version linked list. The version information includes the version number and operation type when the transaction is used to operate on the data. When the transaction is used to operate on the data, before the transaction is committed, the version number is the transaction ID. When the transaction is committed, the version number is updated to the transaction timestamp. The operation optimization module is used to perform the following operations: performing data operations on the data in the column-based database based on the submitted transactions, performing the operations according to the comparison results between the version number and the transaction timestamp, and storing the data before the modification in the undo buffer, wherein the data operations include scan operations, insert operations, delete operations, and update operations; The transaction conflict handling module is used to perform the following: when operating on data based on a transaction, detecting and handling transaction conflicts based on preset conflict handling rules; The storage space management module is used to perform the following operations: divide the storage of the column-based database into table storage and local storage, cache relevant data in the local storage before the transaction is committed, and merge the locally stored data into the table for persistence after the transaction is committed; The version chain table management module is used to perform the following: scan the version information in the version chain table through a version cleanup task, and delete the version information that is no longer referenced by any active transaction based on the transaction status and version information.

2. The system for supporting multi-version concurrency control in a column-based database according to claim 1, characterized in that: When performing a scan operation, the operation optimization module follows the following rule: when the version number is greater than the timestamp of the current transaction and the version number is not equal to the transaction ID of the current transaction, the data corresponding to the version before the transaction is modified is selected.

3. The system for supporting multi-version concurrency control in a column-based database according to claim 1, characterized in that: When performing a modification operation, the data in the original data segment is not modified. A dummy node is reserved at the head of the version linked list to save the updated data. When the data is changed, the modified value is saved directly in the dummy node at the head of the linked list, and the data before the change is saved in the undo buffer. At the same time, the buffer pointer and version information are inserted into the head of the version linked list.

4. The system for supporting multi-version concurrency control in a column-based database according to claim 1, characterized in that: In the distributed database KaiwuDB, each database table has two storage parts: table storage and local storage. When performing an insert operation, the operation optimization module is used to perform the following operations: For the data to be inserted, the row group to be added, the columns in the row group, and the data segments in the columns are found. Based on the compression method used by the data segment, the corresponding compression algorithm is called to add the data to the data segment. When adding data to a data segment, if the data segment space is insufficient, a temporary segment is created. When a row group is full, the data segment is flushed to disk and marked as added to the table. During a rollback, the area on disk storing data is marked as unused. When a transaction is committed, storage commit, undo buffer commit, and WAL flush operations are performed.

5. The system for supporting multi-version concurrency control in a column-based database according to claim 1, characterized in that: When performing a delete operation, the operation optimization module is used to perform the following operations: For the data to be deleted, determine whether the data is stored in the transaction local or in the table by checking whether the row number of the data is greater than MAX_ROW_ID; When deleting data, the version information of the corresponding data is marked as the transaction ID of the current transaction, and a corresponding entry is added to the undo buffer to record the deleted row number; When a transaction is committed, storage commit, undo buffer commit, and WAL flush operations are performed, and the version information in the table is updated to the transaction timestamp.

6. The system for supporting multi-version concurrency control in a column-based database according to claim 1, characterized in that: When performing a modification operation, the operation optimization module is used to perform the following operations: For the data to be modified, determine the data location by its row ID and find the corresponding row group and column data; When modifying the data to be modified, save the data before modification to the undo buffer, modify the data locally, and insert the modified information of the data into the corresponding version list; When a transaction is committed, storage commit, undo buffer commit, and WAL flush operations are performed.

7. The system for supporting multi-version concurrency control in a column-based database according to claim 1, characterized in that: When performing a scanning operation, the operation optimization module is used to perform the following operations: For the data to be scanned, first scan the table, then scan the local storage. When scanning the row group, compare the version numbers corresponding to the insert operation and the version numbers corresponding to the delete operation with the timestamp of the current transaction to determine which data is visible to the current transaction. If the version number corresponding to the data when the insert operation is performed is earlier than the timestamp of the current transaction, and the version number corresponding to the data when the delete operation is performed is later than the timestamp of the current transaction, or the data has not been deleted, then all the data is considered visible. If all data is visible and there are no filtering conditions, directly read the data in each column. If all data is visible but there are filtering conditions, filter according to the filtering conditions and then read the corresponding column data.

8. The system for supporting multi-version concurrency control in a column-stored database according to any one of claims 4 to 7, characterized in that: The transaction conflict handling module is used to perform the following operations: before modifying data based on the current transaction, detect whether other transactions are modifying the data; if so, according to a preset strategy, the current transaction waits for a predetermined time and then attempts to modify the data again; if other transactions complete the read operation on the data after waiting for the predetermined time, the data is modified based on the current transaction; if the data cannot be modified based on the current transaction after waiting for the predetermined time, the current transaction is rolled back.

9. A method for supporting multi-version concurrency control in a column-based database, characterized in that: The steps include: Multi-version management: A transaction ID and timestamp are created for each transaction. A version list is maintained for each data segment in the column-based database. Version information is stored in nodes in the version list. This information includes the version number and operation type for transactions performed on data. Before a transaction is committed, the version number is the transaction ID. When the transaction is committed, the version number is updated to the transaction timestamp. Operation optimization: Data operations are performed on data in the column-based database based on committed transactions. Data operations are performed based on the comparison results between the version number and the transaction timestamp, and the data before modification is stored in the undo buffer. Data operations include scan operations, insert operations, delete operations, and update operations. Transaction conflict handling: When operating on data based on transactions, transaction conflicts are detected and handled based on preset conflict handling rules; Storage space management: The column-based database is divided into table storage and local storage. Before a transaction is committed, relevant data is cached in the local storage. After the transaction is committed, the local storage data is merged into the table for persistence. Version list management: Scan the version information in the version list through the version cleanup task, and delete the version information that is no longer referenced by any active transaction based on the transaction status and version information.

Images